Envoy, Nginx, Apache HTTP Structured Logs with Google Cloud Loggingsalmaan rashidBlockedUnblockFollowFollowingJan 2Google Cloud Logging provides several plugins that allows you to easily emit structured logs for common applications.
For example, if you install the Stackdriver Logging agent, you can get logs using the following fluentd pluginsThis sample demonstrates two of these plugins (apache and nginx) and how to configure them to emit not just structured JSON logs but as a specific HttpRequest protocol buffer.
This article also describes how to configure Envoy proxy for similar httpRequest logging.
The script below sets up the full sample stack with Google Cloud L7 HTTPS loadbalancer and a managed instance group running the webserver.
It does not demo Envoy with the L7 but you can set that up pretty easily with the template below.
In addition to the HttpRequest protocolbuffer, this example and configuration also includes the following in the emitted LogEntrytrace: This is the X-Cloud-Trace-Context header sent via the L7 Loadbalancer.
see payload configurationsspanId: Inbound requests from Google L7 LB includes trace and span to cover.
httpRequest.
latency: Latnecy value provided by nginx/apache and set as the latency` fieldX-Forwarded-For Header is logged when available.
That header is emitted by the L7 LB to indicate the actual originating IP address for the request.
I've intentionally left the clientIp filed for the HttpRequest as the one provided though as a followup TODO: you are welcome to replace that value with the one extracted from this header.
Note, Stackdriver doens’t by default support all fluentd plugins but just the ones listed above.
You are free to customer and define your own plugin described in the documentation in the following article auditd agent config for Stackdriver LoggingFor Reference:Stackdriver Logging AgentStackdriver Structured LoggingStackdriver Logging LogEntryhttp_request.
protoGoogle Cloud Logging plugin for fluentdGetting more value from your Stackdriver logs with structured dataUsecasesFirst, i’ll just show what this will look like in nginx-access logs:Filter by latencyNow that the logs are structured, we can apply a direct filter on those fields.
For example, the following shows which requests took more than 2s to respond:resource.
type="gce_instance"logName="projects/mineral-minutia-820/logs/nginx-access"httpRequest.
latency>"2s"Trace Context loggingEach request in this specific example traverses Google L7 loadbalancer which also get logged.
What that means is you can now trace a request in the LB logs down to the specific instnace in the managed group that handled the call.
For example, if you start with a traceID from the nginx logsnginx LogEntry: trace: "projects/mineral-minutia-820/traces/ba90fb6215439a958799f631ea63b24f"You can find its corresponding entry in the LB LogEntryLB LogEntry: trace: "projects/mineral-minutia-820/traces/ba90fb6215439a958799f631ea63b24f"or run a query like this to cover both.
resource.
type="gce_instance" OR resource.
type="http_load_balancer"trace="projects/mineral-minutia-820/traces/ba90fb6215439a958799f631ea63b24f"Combined LogViewerNow that the frontend server is logging the inital httpRequest as well as the trace/spanId, a backend application that emits the traceId in a log line will result in log grouping.
What that means is the inital request and application logs are grouped in a parent/child format as described in the following articles:Combining correlated Log Lines in Google StackdriverAnd just for reference (and becasue i authored it, :), here is the same impelmented within a webframework (Flask), directlyhttps://github.
com/salrashid123/flask-gcp-log-groupsThis article does not demonstrate this capability but if its useful, i can be convinced to provide an example.
Filter by SourceIPIf requests traverse the Google Loadbalancer, it injects the standard X-Forwarded-For header which includs the actual source IP address as well as the proxies this request traversed.
In the case for Google, it will include the incident SSL Proxy.
For examplex_forwarded_for: "73.
162.
112.
208, 35.
190.
3.
148"The origin ip is actually 73.
162.
112.
208.
Currently, the parsers just uses the provided value as the clientIP which is not the derived origin value (its still the loadbalancer).
If you want to parse the actual origin IP, you will need to parse out the value from the header and add it to for the remoteIP.
For apache, its something like this:parser_apache.
rb:host = m['host'].
delete(' ')host = host.
split(',').
firstapache.
conf<record> .
httpRequest ${ {.
.
"remoteIp" => record['host'] .
.
} .
</record>You can find the full source here:salrashid123/gcp_envoy_nginx_apache_structured_logsEnvoy, Nginx, Apache HTTP Structured Logging with Google Cloud Logging …github.
comNGINXIf you prefer to use nginx, you can create the full LB->nginx managed instance group by running the following commands in sequnce:CreateIf can use the certs provided in this repo but if you prefer to setup your own…openssl req -x509 -newkey rsa:2048 -keyout key.
pem -out cert.
pem -days 365 –nodes -subj "/C=US/ST=California/L=San Francisco/O=Google/OU=Cloud/CN=server.
somedomain.
com"then finish off the step:gcloud compute ssl-certificates create testcert –certificate=cert.
pem –private-key=key.
pemgcloud alpha compute target-https-proxies create nginx-lb-proxy –url-map=nginxpool-map –ssl-certificates=testcert –globalgcloud compute forwarding-rules create nginx-content-rule –global –target-https-proxy nginx-lb-proxy –ports 443You should end up with an L7 LB pointing to an instance group with four serversNOTE: it may take upto 10 minutes for the L7 LB to provide the initial provisioning so feel free to get a coffee nowTestOnce the LB is seutp you can send traffic downexport GATEWAY_IP=$(gcloud compute forwarding-rules describe nginx-content-rule –format="value(IPAddress)" –global)echo $GATEWAY_IPfor i in {1.
100}; do curl -o /dev/null -sk -w "%{time_total}." https://$GATEWAY_IP/backend; sleep 1; done2.
7559320.
7086674.
7582572.
2510392.
7030432.
5425904.
6039074.
555793The responses above indicate how long the specific backend took.
As you see, since we applied a variable delay into each instance as part of the startup script, each VM will take a minimum amount of time to respondDeleteTo delete the cluster, run the following in sequence:ConfigThe following details the configurations used.
Eventually, this should also get rolled into the standard google-fluentd library so wont' be a need to add the custom code below.
nginx/etc/nginx/nginx.
conf/etc/nginx/sites-enabled/defaultgoogle-fluentd/opt/google-fluentd/embedded/lib/ruby/gems/2.
4.
0/gems/fluentd-1.
2.
5/lib/fluent/plugin/parser_nginx.
rb/etc/google-fluentd/config.
d/nginx.
confApache2If you use Apache2, you can create the full stack by running the commands below in sequence.
Createhttps://gist.
github.
com/salrashid123/5f1a2ddc74599611698778836dfb25d2You can use the certs provided in this repo but if you prefer to setup your own…openssl req -x509 -newkey rsa:2048 -keyout key.
pem -out cert.
pem -days 365 –nodes -subj "/C=US/ST=California/L=San Francisco/O=Google/OU=Cloud/CN=server.
somedomain.
com"thengcloud compute ssl-certificates create testcert –certificate=cert.
pem –private-key=key.
pemgcloud alpha compute target-https-proxies create apache-lb-proxy –url-map=apachepool-map –ssl-certificates=testcert –globalgcloud compute forwarding-rules create apache-content-rule –global –target-https-proxy apache-lb-proxy –ports 443DeleteTestAs with the example with nginx, you can send requests directly via curl in loop and gauge response times.
export GATEWAY_IP=$(gcloud compute forwarding-rules describe apache-content-rule –format="value(IPAddress)" –global)echo $GATEWAY_IPfor i in {1.
100}; do curl -o /dev/null -sk -w "%{time_total}." https://$GATEWAY_IP/backend; sleep 1; doneConfigThe following details the configurations used for apache.
Eventually, this should also get rolled into the standard google-fluentd library so wont' be a need to add the custom code below.
Apache2/etc/apache2/apache2.
conf/etc/apache2/sites-enabled/000-default.
confgoogle-fluentd/opt/google-fluentd/embedded/lib/ruby/gems/2.
4.
0/gems/fluentd-1.
2.
5/lib/fluent/plugin/parser_apache2.
rb/etc/google-fluentd/config.
d/apache.
confEnvoyEnvoy Access logs is fairly customizable and can write to any number of targets.
For example, you can configure envoy to emit logs to remotely (see envoy_control#accesslog) or in this article, locally to a log file where Cloud Logging can do the rest of the legwork.
This article includes two separate configurations for envoy and google-fluentd: Default envoy settings and one optimized for GCP that includes the cloud-trace-context header.
defaultThe steps to emit default logs is fairly easy: just set the path where the logs will write to- name: envoy.
http_connection_manager config: access_log: – name: envoy.
file_access_log config: path: "/tmp/envoy.
log"Custom header (cloud-trace-context)You can customize envoy’s logs easily by adding in fields like custom headers into the log file.
The following shows a snippet on now to setup the logs that extend the default configuration and just adds %REQ(X-Cloud-Trace-Context)%- name: envoy.
http_connection_manager config: access_log: – name: envoy.
file_access_log config: path: "/tmp/envoy.
log" format: "[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" "%REQ(X-Cloud-Trace-Context)%"."TestingTo test this, you can incorporate the envoy configurations provided into a similar VM as described above.
Remember to copyparser_envoy.
rb –> /opt/google-fluentd/embedded/lib/ruby/gems/2.
4.
0/gems/fluent-plugin-google-cloud-0.
7.
2/lib/fluent/plugin/out_google_cloud.
rbenvoy.
conf –> /etc/google-fluentd/config.
d/envoy.
confIf you are on a VM with google-fluentd and the configurations settings above, restart fluentd and run envoy:.
/envoy -l info -c envoy_config.
yamlEither send in a request via the LB or directly to test:curl -H "X-Cloud-Trace-Context: 1212/bbb" -H "X-Forwarded-For: 12.
3.
4.
5, 1.
2.
3.
4" http://localhost:10000In cloud logging, you will see the standard httpRequest payload as well as envoy specific headers:From there, you can further customize the headers you want to caputure by modifying the fluentd and envoy configurations.
Thats all folks.