AWS Security Lake gives you a place to put security logs: S3, in OCSF, catalogued in Glue and queryable from Athena. What it does not give you is a way to get your non-AWS logs into it, or anything that tells you when something in there matters.

This post covers how we built those two halves: ingest and alerting. A third topic runs through both, and it is the part I would most like other teams to copy: authentication. That covers how the ingest authenticates to the outside world, and how anyone reading the lake authenticates to it.

What the lake is for

First, a framing that shaped every decision below. The lake is a query surface, not the system of record. Our authoritative audit trail is the organisation CloudTrail, delivered into an isolated, validated, KMS-encrypted vault account. If the lake and the vault disagree, the vault wins.

So the lake can make trade-offs the system of record cannot, such as hourly latency, re-cutting a table, or dropping a mapping and replaying it. What it must not do is fail silently.

Ingest

Native sources

The AWS-native sources come for free once Security Lake is enabled: CloudTrail management events, Lambda execution events, EKS audit logs, VPC flow logs, Route 53 resolver logs and Security Hub findings, for every account in the organisation. They land as Iceberg tables. Since a September rollup, our us-east-1 data is queryable from Sydney as well.

Custom sources

Everything else is a custom source: a Lambda that fetches or receives events, maps them to OCSF, writes Parquet into the lake’s bucket, and starts the Glue crawler. Four ingests feed seven custom sources:

Source	How it arrives	OCSF classes
Auth0 (both portal tenants)	Pushed: SNS → SQS, drained hourly	Authentication, API Activity
GitHub enterprise audit log	Pushed: GitHub streams to S3, keys queued, drained hourly	Authentication, API Activity
Entra ID sign-ins and audits	Pulled from Microsoft Graph	Authentication, Account Change
Entra ID Protection and Defender XDR	Pulled from Microsoft Graph	Detection Finding, Incident Finding
Defender device vulnerabilities	Pulled from Microsoft Graph advanced hunting	Vulnerability Finding
Athena query history	Pulled from Athena daily	API Activity

They live in their own repository, separate from the rest of our security infrastructure, for one reason: tests. OCSF mappings rot silently. Microsoft adds an @odata.type, a field changes shape, and nothing tells you until an investigation comes back empty. Every mapping is written to Parquet in CI and read back with its schema compared. The test fixtures are real events from the live feeds with identifiers replaced, not examples from vendor docs, because the docs do not say which fields a tenant actually sends.

That paid off immediately. Our original GitHub mapping modelled eight fields. The payload carries about ninety. Every one of 2.77 million rows said its activity was “Create” and its status was “Success”, so the table could not answer “what was changed” or “what was refused”. The Auth0 mapping was quietly dropping geolocation, TLS fingerprints (JA3/JA4) and the customer organisation, and nothing put them even in unmapped.

Design rules we settled on

Structs, not JSON strings. Our first custom sources serialised the nested OCSF objects with json.dumps, so the crawler catalogued them as strings and every query needed json_extract_scalar. The native tables are structs. A cross-source join therefore switched idioms mid-query, and Athena had to parse a whole blob per row. Now actor, src_endpoint, metadata and the rest are real Parquet structs. Only genuinely polymorphic fields such as unmapped and evidences stay as JSON strings. We also spell location fields identically across Auth0, Entra and GitHub, so “the same person, three systems” is a single query.

Batch hourly, not per event. Invoked per S3 object, the GitHub transform wrote about 10,800 Parquet files a day for about 33,000 events, at roughly 5 KB per event, most of it Parquet footer. Athena scans small files slowly long before they become expensive. Draining a queue hourly brings that to about 24 files a day. The cost is that an event is queryable within the hour instead of within minutes. That is the right trade, because real-time paging is not the lake’s job.

Know whether your ingest can replay. This decides most of the failure handling:

• Graph and Athena history can be asked again, so those ingests rewrite whole days under deterministic keys. A retry replaces its own object, and a missed run backfills itself.
• An SQS queue can only be drained once, so the Auth0 ingest writes before it deletes. It names objects after the scheduler’s fire time so a retry overwrites instead of duplicating. It also reads the Failed list from DeleteMessageBatch, because that call can partially succeed while returning normally.
• GitHub’s raw bucket is the system of record for that feed: no lifecycle policy, and a writer that can only PutObject. The queue carries keys, not events, so a lost message is a gap we can refill by re-enqueueing. When we re-cut that table, we replayed the outage window from the bucket and lost nothing.

One bad record must not stall the feed. An unparseable message is skipped and counted, then left to dead-letter on its own. Its log line carries the message id and size and nothing else, because an Auth0 Management API event contains a whole user profile and CloudWatch is not where that belongs. A test asserts nothing else reaches stdout.

Restart the crawler yourself. One surprise: when you register a custom source, Security Lake gives its crawler a one-off cron schedule that fires on that day only. All three of our re-registrations got one. Each drain therefore calls StartCrawler itself, and that call is the only thing keeping the schema current.

Authentication

The rule: no long-lived external credential at rest in the security account. Every source authenticates without a stored secret, and every reader authenticates as a named human.

This rule was set in review, when the Microsoft Graph ingest first proposed a client secret. It would have been the first long-lived external credential in the account, a string that can be copied out of a log and that expires on a 24-month timer. Here is how each source avoids one.

Inbound: three sources, three patterns

GitHub pushes to us over OIDC. GitHub Enterprise Cloud streams its audit log straight into an S3 bucket. It authenticates by assuming an IAM role through an AWS::IAM::OIDCProvider that trusts GitHub’s issuer. The role can PutObject and nothing else: it cannot read, list or delete. A compromised stream therefore cannot read the audit log back out or destroy what is already there. Agent and human read roles are also denied GetObject on that raw bucket, since it holds the unredacted log.

Auth0 arrives by SNS, across accounts. An Auth0 log stream can target exactly one AWS account and region, and our tenants already stream to their own application accounts. Those accounts publish every event to an SNS topic. The security account subscribes an SQS queue to both topics. Authentication here is a topic policy that allows the cross-account subscription, so there is no API key and nothing polls Auth0.

Microsoft Graph: we became an OIDC issuer. This one is the most interesting. Graph will not answer without an app credential. Entra, however, supports workload identity federation: it accepts a signed assertion from any OIDC issuer that publishes a discovery document and a JWKS over public HTTPS. Lambda has no identity token of its own, so we made the account an issuer:

1. A non-exportable RSA key in KMS per workload signs a ten-minute JWT assertion.
2. A small publisher Lambda lists every signing key under a KMS alias prefix. It writes their public halves into one JWKS, served from a private S3 bucket behind CloudFront along with the discovery document.
3. Entra holds a federated credential that pins that issuer URL and the workload’s subject. It exchanges the assertion for a Graph token.

This is the mirror image of the GitHub setup. There, AWS trusts someone else’s issuer. Here, someone else trusts ours. The security question changes from “who can read the secret” to “who can call kms:Sign on that key”. CloudTrail records that permission against a caller, and a key policy change revokes it at once.

A few details we learned along the way:

• The publisher refuses to publish an empty key set. A revoked permission looks exactly like “no keys”, and publishing that would break every federated workload at once.
• kms:GetPublicKey is scoped by kms:RequestAlias, so the publisher can read only keys under its prefix.
• An authentication failure is fatal to the run, not logged and skipped. If the trust breaks on the Microsoft side, the run alarms instead of quietly writing zero rows.
• The issuer bucket holds nothing else. It is the only bucket in the account behind a public distribution.
• Use a domain you own. The iss claim is compared exactly, and we used the CloudFront default domain. Replacing the distribution would mean updating every federated credential that trusts it.

Outbound: who can read the lake

Reading is the other half of authentication, and Lake Formation has one property that has cost us more time than anything else:

Lake Formation hides what you have not been granted. It does not deny it. A missing grant shows up as an empty result set or an empty SHOW DATABASES, with no error anywhere.

IAM lakeformation:GetDataAccess only lets a role ask. Lake Formation still decides, and it answers “no” with silence. A new source, a recreated role or a dropped grant all look exactly like an ingest that wrote nothing.

We handle that in three ways:

• Humans and agents use dedicated read-only identities. Investigations, including those run by AI agents, use an Identity Center permission set separate from anyone’s browser role. It is read-only, has no console access, and explicitly denies secret values, kms:Decrypt and S3 object contents outside the lake. Its one write is the Athena results prefix. Every call lands in CloudTrail as the human who signed in, and widening its access is a pull request.
• The detection runner’s grant is in CloudFormation. It is the only Lake Formation grant we manage in a template, because of how it fails without one: a runner that lost its grant would keep running hourly, keep succeeding and alert on nothing. The role also has no S3 read on the lake bucket. Rows arrive through Lake Formation credential vending, and a direct s3:GetObject would be a bypass that kept working after the grant was revoked.
• Every query is itself audited. CloudTrail records who called StartQueryExecution, but it replaces the SQL with ***OMITTED***, and that is not configurable. Athena keeps the text for 45 days and never exports it. So one of our custom sources copies Athena’s query history into the lake, and a join to CloudTrail on the execution id attributes each query to a person. The unit of accountability is the human identity, not the tool. Console, CLI and MCP-server queries all land in the same place. One trap: on the CloudTrail side, the execution id is in api.response.data, not api.request.uid. The latter is populated, looks plausible and silently joins to nothing.

Alerting

We split alerting into two kinds, and keeping them separate made both simpler.

Pipeline alarms: did the data arrive?

Every ingest stack carries CloudWatch alarms on its own plumbing: Lambda errors, dead-letter depth, queue age, and “has not run in three hours” (with TreatMissingData: breaching). None of these looks at an event. They all publish to one alarm topic shared by every stack, so no stack ends up with a topic nobody subscribes to.

One check is worth doing after any deploy that recreates a topic. An SNS email subscription sits in PendingConfirmation until someone clicks the link, while every alarm deploys cleanly without it. The stack looks monitored and delivers nothing.

Detections: what does the data say?

Content alerting is a separate stack, a scheduled SQL rule engine:

EventBridge (hourly) → Lambda → Athena, one query per rule → Security Hub → EventBridge → SNS

Each rule is one .sql file with a header giving its severity, window and the tables it reads. There are fourteen today, in roughly three groups:

• Identity: a sign-in from outside Australia (Entra and Auth0), failures clustered in one /24, Entra ID Protection flagging a staff account, and a password spray followed by a success from the same subnet.
• GitHub: a PAT with write scope authorised against the org, the first successful use of a token with admin:org or delete_repo, a repository transferred out, enterprise admin changes, and an actor’s first SSH push in 30 days.
• The lake itself: the runner cannot see any CloudTrail, a machine-driven custom source has stopped writing partitions, or the lake was queried by a principal who is not a known analyst.

Why scheduled queries and not a streaming detector?

1. Latency is already capped by ingest. No detector can see an event before the hourly drain writes it.
2. Only a query can ask the interesting questions. Thresholds, GROUP BY, cross-source joins, and above all absence. “No Auth0 events in six hours” is a real detection, and a streaming detector cannot see something that did not happen.
3. Rules become testable. CI resolves every table and column path a rule reads against a pinned snapshot of the live Glue catalog. A mapping change that renames a field then fails the build, instead of leaving a rule that matches zero rows forever.

Why Security Hub and not straight to SNS? The delivery path from GuardDuty through EventBridge already existed and was proven. A stable finding Id makes a re-import an update, not a duplicate. Best of all, Security Hub findings are themselves a Security Lake source, so a detection flows back into the lake and can be queried next to the events that triggered it.

Deduplicating findings is not deduplicating email

Security Hub folds a repeated Id into one finding. But every BatchImportFindings call emits a “Findings - Imported” event, updates included, so each re-import still sends an email. Our rule windows deliberately overlap the schedule so a skipped run does not drop events. Left alone, a six-hour window meant six emails per event. A source-down rule with no window at all would have emailed hourly for the length of the outage: about seventy emails over a weekend.

So the runner asks Security Hub what it already holds open and imports only what is new or has come back. Resolved findings that recur reopen and notify again. Finding keys never contain a timestamp or a count. Most use a day, which is coarse enough to survive a sliding window and fine enough that next month’s recurrence is a new finding. The source-down rule uses no time at all, because it describes a condition that persists. A test pins this.

Tuning against real data

Running rules against real data was humbling. Three rules were wrong in the same direction: they assumed Auth0 was a staff directory, when it is our customer portal.

• The overseas-login rule fired on a client’s employee signing in from Japan. It is now scoped to staff accounts.
• The spray rule counted failures. On its first run it reported two colleagues behind one office NAT as compromised: one had mistyped a password four times, the other had failed nothing. A spray is one source trying many accounts, so the rule now counts distinct usernames. That threshold is silent across the table’s entire history, and a real spray clears it immediately.
• Cross-source rules hit two silent traps. Entra puts the identity in user.name and GitHub in actor.user.name. Auth0 leaves the country null on roughly half of successful logins, so country <> 'AU' quietly drops them.

Two rules were written and then deleted. A “new TLS fingerprint for this user” rule turned out to detect browser upgrades. A branch-protection rule alerted on the control working. A detection set is only worth reading if every finding is.

Watching the watcher

The hardest alert to get right was the one that says the alerting itself is broken.

The first attempt raised an error whenever every rule returned zero rows, on the theory that a lost Lake Formation grant shows up as empty results. The first deploy disproved it: all rules returning zero is the normal state, because these are rules about things that should not happen. Absence is the expected result, so it cannot also be the fault signal.

The replacement is a positive control. One rule fires when the runner can see no CloudTrail at all. CloudTrail depends on none of our ingests and writes about 24,000 rows an hour, so zero means the grant is gone or collection stopped.

Source freshness taught a similar lesson. Measure it by the newest partition, not the newest event time. Our Graph ingest rewrites whole days, so a week-old risk detection is rewritten today with last week’s timestamp. And keep sources driven by people out of the freshness rule altogether: Athena query history legitimately writes nothing on a weekend. Those sources rely on their own “has not run” alarms.

What I would tell someone starting

1. Decide early what is the system of record, so the lake is free to be re-cut.
2. Make external auth secretless from day one. OIDC in, OIDC out, and cross-account SNS. A KMS key you can revoke beats a secret you have to rotate.
3. Treat empty results as suspect. Lake Formation fails silently, so build a positive control.
4. Map from real payloads and test the schema in CI. Vendor docs will not tell you what your tenant sends.
5. Keep pipeline health and content detection apart. Each is simple alone and confusing together.