Affecting all Beats

• The document id fields has been renamed from @metadata.id to @metadata._id 15859

• Variable substitution from environment variables is not longer supported. https://github.com/elastic/beats/pull/15937{15937}

• Change aws_elb autodiscover provider field name from elb_listener.* to aws.elb.*. 16219 https://github.com/elastic/beats/pull/16402{16402}

• Remove AddDockerMetadata and AddKubernetesMetadata processors from the script processor. They can still be used as normal processors in the configuration. 16349 16514

Auditbeat

• File integrity dataset (macOS): Replace unnecessary file.origin.raw (type keyword) with file.origin.text (type text). 12423 15630

Filebeat

Heartbeat

Journalbeat

• Improve parsing of syslog.pid in journalbeat to strip the username when present 16116

Metricbeat

• Make use of secure port when accessing Kubelet API 16063

• Add Tomcat overview dashboard 14026

Packetbeat

Winlogbeat

Functionbeat

Affecting all Beats

• Fix add_cloud_metadata to better support modifying sub-fields with other processors. 13808

• TLS or Beats that accept connections over TLS and validate client certificates. 14146

• Fix panics that could result from invalid TLS certificates. This can affect Beats that connect over TLS, or Beats that accept connections over TLS and validate client certificates. 14146

• Fix panic in the Logstash output when trying to send events to closed connection. 15568

• Fix missing output in dockerlogbeat 15719

• Fix logging target settings being ignored when Beats are started via systemd or docker. 12024 15442

• Do not load dashboards where not available. 15802

• Fix issue where TLS settings would be ignored when a forward proxy was in use. https://github.com/elastic/beats/pull/15516{15516}

• Update replicaset group to apps/v1 15802

• Fix issue where default go logger is not discarded when either * or stdout is selected. 10251 15708

• Upgrade go-ucfg to latest v0.8.1. https://github.com/elastic/beats/pull/15937{15937}

• Fix index names for indexing not always guaranteed to be lower case. 16081

• Add ssl.ca_sha256 option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. 15717

• Fix loading processors from annotation hints. 16348

• Fix an issue that could cause redundant configuration reloads. 16440

• Fix k8s pods labels broken schema. 16480

• Fix k8s pods annotations broken schema. 16554

• Upgrade go-ucfg to latest v0.8.3. https://github.com/elastic/beats/pull/16450{16450}

• Fix NewContainerMetadataEnricher to use default config for kubernetes module. 16857

• Improve some logging messages for add_kubernetes_metadata processor 16866

• Fix k8s metadata issue regarding node labels not shown up on root level of metadata. 16834

Auditbeat

• system/socket: Fixed compatibility issue with kernel 5.x. 15771

Filebeat

• Ensure all zeek timestamps include millisecond precision. 14599 16766

• Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. 15502 15590

• Add shared_credential_file to cloudtrail config 15652 15656

• Fix typos in zeek notice fileset config file. 15764 15765

• Fix mapping error when zeek weird logs do not contain IP addresses. 15906

• Improve elasticsearch/audit fileset to handle timestamps correctly. 15942

• Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the elasticsearch module. 15840 15900

• Fix mapping error for cloudtrail additionalEventData field 16088

• Fix a connection error in httpjson input. 16123

• Fix s3 input with cloudtrail fileset reading json file. 16374 16441

• Rewrite azure filebeat dashboards, due to changes in kibana. 16466

• Adding the var definitions in azure manifest files, fix for errors when executing command setup. 16270 16468

• Fix merging of fileset inputs to replace paths and append processors. https://github.com/elastic/beats/pull/16450{16450}

• Add queue_url definition in manifest file for aws module. https://github.com/elastic/beats/pull/16640{16640}

• Fix issue where autodiscover hints default configuration was not being copied. 16987

Heartbeat

• Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. 15639

Journalbeat

Metricbeat

• Add dedot for tags in ec2 metricset and cloudwatch metricset. 15843 15844

• Use RFC3339 format for timestamps collected using the SQL module. 15847

• Avoid parsing errors returned from prometheus endpoints. 15712

• Change lookup_fields from metricset.host to service.address 15883

• Add dedot for cloudwatch metric name. 15916 15917

• Fixed issue logstash-xpack module suddenly ceasing to monitor Logstash. 15974 16044

• Fix skipping protocol scheme by light modules. pull

• Made logstash-xpack module once again have parity with internally-collected Logstash monitoring data. 16198

• Change sqs metricset to use average as statistic method. 16438

• Revert changes in docker module: add size flag to docker.container. 16600

• Fix diskio issue for windows 32 bit on disk_performance struct alignment. 16680

• Fix detection and logging of some error cases with light modules. 14706

• Fix imports after PR was merged before rebase. 16756

• Add dashboard for redisenterprise module. 16752

• Dynamically choose a method for the system/service metricset to support older linux distros. 16902

Packetbeat

• Enable setting promiscuous mode automatically. 11366

Winlogbeat

Functionbeat

• Fix timeout option of GCP functions. 16282 16287

Affecting all Beats

• Add document_id setting to decode_json_fields processor. 15859

• Include network information by default on add_host_metadata and add_observer_metadata. 15347 16077

• Add aws_ec2 provider for autodiscover. 12518 14823

• Add monitoring variable libbeat.config.scans to distinguish scans of the configuration directory from actual reloads of its contents. 16440

• Add support for multiple password in redis output. 16058 16206

• Add support for Histogram type in fields.yml 16570

• Windows .exe files now have embedded file version info. 15232t

• Remove experimental flag from setup.template.append_fields 16576

• Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. 16621

• Add Kerberos support to Kafka input and output. 16781

• Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. https://github.com/elastic/beats/pull/16621[16621

• Add support for kubernetes provider to recognize namespace level defaults 16321

• Add translate_sid processor on Windows for converting Windows security identifier (SID) values to names. 7451 16013

Auditbeat

Filebeat

• Set event.outcome field based on googlecloud audit log output. 15731

• Add dashboard for AWS ELB fileset. 15804

• Add dashboard for AWS vpcflow fileset. 16007

• Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb 15757 15936

• Add custom string mapping to CEF module to support Forcepoint NGFW 14663 15910

• Add ingress nginx controller fileset 16197

• move create-[module,fileset,fields] to mage and enable in x-pack/filebeat 15836

• Add ECS tls and categorization fields to apache module. 16032 16121

• Add MQTT input. 15602 16204

• Add ECS categorization fields to activemq module. 16151 16201

• Add a TLS test and more debug output to httpjson input 16315

• Add an SSL config example in config.yml for filebeat MISP module. 16320

• Improve ECS categorization, container & process field mappings in auditd module. 16153 16280

• Improve ECS field mappings in aws module. 16154 16307

• Improve ECS categorization field mappings in googlecloud module. 16030 16500

• Improve ECS field mappings in haproxy module. 16162 16529

• Add cloudwatch fileset and ec2 fileset in aws module. 13716 16579

• Improve ECS categorization field mappings in kibana module. 16168 16652

• Improve the decode_cef processor by reducing the number of memory allocations. 16587

• Add cloudfoundry input to send events from Cloud Foundry. 16586

• Improve ECS categorization field mappings in iis module. 16165 16618

• Improve ECS categorization field mapping in kafka module. 16167 16645

• Allow users to override pipeline ID in fileset input config. 9531 16561

• Add o365audit input type for consuming events from Office 365 Management Activity API. 16196 16244

• Improve ECS categorization field mappings in logstash module. 16169 16668

• Update filebeat httpjson input to support pagination via Header and Okta module. 16354

• Improve ECS categorization field mapping in icinga module. 16164 16533

• Improve ECS categorization field mappings in ibmmq module. 16163 16532

• Improve ECS categorization, host field mappings in elasticsearch module. 16160 16469

• Add ECS related fields to CEF module 16157 16338

• Improve ECS categorization field mappings in suricata module. 16181 16843

Heartbeat

• Allow a list of status codes for HTTP checks. 15587

Journalbeat

Metricbeat

• Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. 15503

• Add lambda metricset in aws module. 15260

• Expand data for the system/memory metricset 15492

• Add azure storage metricset in order to retrieve metric values for storage accounts. 14548 15342

• Add cost warnings for the azure module. 15356

• Add DynamoDB AWS Metricbeat light module 15097

• Release elb module as GA. 15485

• Add a system/network_summary metricset 15196

• Add mesh metricset for Istio Metricbeat module 15535

• Add mixer metricset for Istio Metricbeat module 15696

• Add pilot metricset for Istio Metricbeat module 15761

• Make the system/cpu metricset collect normalized CPU metrics by default. 15618 15729

• Add galley metricset for Istio Metricbeat module 15857

• Add key/value mode for SQL module. 15770 {pull]15845[15845]

• Add STAN dashboard 15654

• Add support for Unix socket in Memcached metricbeat module. 13685 15822

• Add up metric to prometheus metrics collected from host 15948

• Add citadel metricset for Istio Metricbeat module 15990

• Add support for processors in light modules. 14740 15923

• Add collecting AuroraDB metrics in rds metricset. 14142 16004

• Reuse connections in SQL module. 16001

• Improve the logstash module (when xpack.enabled is set to true) to use the override cluster_uuid returned by Logstash APIs. 15772 15795

• Add region parameter in googlecloud module. 15780 16203

• Add kubernetes storage class support via kube-state-metrics. 16145

• Add database_account azure metricset. 15758

• Add support for NATS 2.1. 16317

• Add Load Balancing metricset to GCP 15559

• Add support for Dropwizard metrics 4.1. 16332

• Add azure container metricset in order to monitor containers. 15751 16421

• Improve the haproxy module to support metrics exposed via HTTPS. 14579 16333

• Add filtering option for prometheus collector. 16420

• Add metricsets based on Ceph Manager Daemon to the ceph module. 7723 16254

• Release statsd module as GA. 16447 14280

• Add collecting tags and tags_filter for rds metricset in aws module. 16605 16358

• Add OpenMetrics Metricbeat module 16596

• Add cloudfoundry module to send events from Cloud Foundry. 16671

• Add redisenterprise module. 16482 15269

• Add system/users metricset as beta 16569

• Align fields to ECS and add more tests for the azure module. 16024 16754

• Add additional cgroup fields to docker/diskiohttps://github.com/elastic/beats/pull/16638[16638]

• Add PubSub metricset to Google Cloud Platform module 15536

• Add Prometheus remote write endpoint 16609

• Release STAN module as GA. 16980

Packetbeat

Functionbeat

Winlogbeat

• Add more DNS error codes to the Sysmon module. 15685

• Add experimental event log reader implementation that should be faster in most cases. 6585 16849

Affecting all Beats

Filebeat

Heartbeat

Journalbeat

Metricbeat

Packetbeat

Winlogbeat

Functionbeat

Journalbeat