Querying security logs in AWS has never been easy. The Cloudtrail console is terrible, VPC and ALB logs are just dumped to S3. Add multiple regions to the mix and it became clear that AWS was concerned about creating a good audit trail, querying that audit trail was somebody elses problem. Any that somebody is you.

AWS security lake provdied a great answer to that problem, set it up in our security via the console and you finally have all your security logs in a querablable format.

But why stop there?

Querying github logs is terrible too.

I have wasted countless hours tracking down and verifying incidents. Github sent an email tell us that a enterprise app needed more permission but when I went to the app it wasn’t asking for anything new. Did another admin already approve, was the app updated? In the end I had to give an internal app an extra permission (and roll back afterwards) so I see what this looked like in the logs. From there I could craft a query to find the enterprise app update and later rollback. Finally we could send our findings to the vender who confirmed it was a legitate push and rollback. It shouldn’t be this hard.

A guardduty alert about a suspious IP address modifying resouces, it was pretty obvious that it was a legit github action deploying a new version of our website but its still necessary to put in the work to which action was running and that the github Azure IP address matched the AWS logs.

These incidents have to be followed up on but it doesn’t have to be so painfully slow. Just add the github logs to security lake.

Any why stop at github logs? Add Entra next and any other vendor logs you have.

In an AI world the actual code for these integrations doesn’t matter, just get claude to write it for you, probably in python. But here is a few points I think you (or your AI agent) should consider when writing the integration.

Github

The first step is to stream the github logs to S3, authenticate by assuming an IAM role through AWS::IAM::OIDCProvider. The role should only have the PutObject permission.

The GitHub ingest writes very small files, 3 or 4 entries per file, using that as an event trigger directly lead to a huge number of Parquet files. Thats an overhead, its an overhead you don’t want, S3 really dooesn’t like small files. Instead use the s3 notification to put a message on an SQS queue, then an hourly cron job to drain, convert and write the Parquet file.

You want to convert the github the important audit fields to Typed Parquet structs, everything else kept in unmapped. That will give you cheaper, faster queries and it fits with how the existing AWS Security Lake tables work (CloudTrail, VPC flow, etc). Out implementation used structs for about 30 of the 90 fields.

GitHub periodically writes a small file called _check to the bucket. This isn’t JSON file and will break your ingest if you don’t tell your coding agent to account for it.

Entra

The Entra integration was an easy add-on after GitHub: the AI had an example to follow, and the Entra Graph API is well documented. And yet I’m finding it hard to explain the setup.

First off, unlike GitHub, which pushes its logs to AWS, you have to pull the logs from Entra. That means authenticating as an Entra enterprise app. You don’t want long-lived keys or secrets; that’s the road to doom.

Instead, your AWS security account can act as an OIDC identity provider. The polling Lambda uses a KMS key to sign a short-lived token, and Entra swaps that for a Graph access token.

Sounds complicated, but it’s not. What you will need is:

• an empty washing-up liquid bottle
• a clean yogurt pot
• some tin foil
• a bit of sticky tape
• an RSA KMS key that can’t be exported (RSA_2048, sign/verify)
• a private S3 bucket, behind CloudFront, serving the two public files Entra reads: – /.well-known/openid-configuration, which tells Entra where the keys are – /.well-known/jwks.json, the public key itself
• a bash script you run once (and again if you ever replace the key). It: – reads the public half of the KMS key – formats it as a JWKS – uploads that as jwks.json, along with the openid-configuration discovery document

The Entra side is easier. Create a single-tenant app with no secret or certificate. Its only credential is a federated credential using the “Other issuer” scenario, which trusts our own OIDC issuer: the CloudFront URL in front of the bucket we just created. Make sure you enter it exactly, with no trailing slash. Adding a trailing slash could cost you hours… or so I’m told. It totally didn’t happen to me.

The audience is api://AzureADTokenExchange, and the subject can be anything, such as security-lake. Just put the same value in the sub claim when the polling Lambda authenticates.

Lastly, give the app the Graph application permissions it needs, such as AuditLog.Read.All for sign-in logs, then grant admin consent, and your auth setup is done.

Thats it, get you coding AI to write the polling lambda and pull in what you need. For example:

• ext_entralogs – /auditLogs/signIns: Every sign-in, with result, IP, location and device – /auditLogs/directoryAudits: Directory changes: users, groups, roles, apps
• ext_entrarisk – /identityProtection/riskDetections: Individual risk events, such as leaked credentials or unfamiliar sign-ins – /identityProtection/riskyUsers: A daily snapshot of who is currently rated risky
• ext_defenderxdr – /security/alerts_v2: Defender XDR alerts – /security/incidents: Defender XDR incidents
• ext_defendervulns – /security/runHuntingQuery: A daily snapshot of CVEs currently exposed on devices

Querying

This is the real game changer. Don’t try to write queries against the lake, ask AI to find the info you need and it writes the queries. It might say SQL on the tin but its still a slog and you are not going to be writing enough queries to get good at it.

Alerting

How that we have a lake, we can also have alerts. The simplest approach is schedule an lambda to run houly that runs queries against the lake. The queries need to look back serveral hours, or even a day so a skipped invocation or an outage won’t miss anything.

Instead of managing state in dynomdb or something similar we turned each alert into a Security Hub finding. The rule needs to set its own finding_id carefully, usually including the day, to ensure we don’t get a duplicate finding when the alerting lambda finds the same row again in its next invocation. The runner asks Security Hub what’s already open and skips them.

Next is an event bridge rule that fires for source: aws.securityhub with a finding generator id prefix that matches your alerts. From there so an SNS topic and then final to email/slack/your on-call alerting platform/whatever.

The queries themselves are just SQL, or what passses for SQL in the Athena world. Here is a quick flavour of what we look out for:

• github-enterprise-admin-added: a new admin is added to github
• github-repo-transfer-outgoing: a repo transferred outside of our enterprise account
• github-dangerous-token-scope: a token with an admin scope is used for the first time (we can’t detect that it was created, so first use will have to do)
• github-pat-authorized-for-org: a write-capable PAT is authorised against the org
• athena-query-by-unknown-principal: someone outside the known analysts queries the lake
• lake-source-silent: a machine-fed source has stopped writing

Who audits the auditor

Speaking of Athena, querying the lake really means running an Athena query against the lake. These are automatically kept by Athena for 45 days and because we are a AWS security account nothing else uses Athen, so they are all our queries.

We need the logs for 365 days, to achieve that we wrote a lambda, it runs daily and copies every query execution from every Athena workgroup into the athena-query-history security lake table.

cloud_trail_mgmt_2_0 already holds the caller. it logs StartQueryExecution with the identity, but not the SQL, that get replaced with OMITTED. Athena’s history is the each opposite, no caller but we get the sql.

One gotta with a daily lamba sync is that it could be 1 day before you can query this but for this one uncase it seemed acceaptable.

Some final words of warning

Try to lock in your schema for each table early, my orginal design brought in brought in the github logs as unstructured data. Whem I finally decided to fix it and move to struts I was left with a dialimia. Do I just add another table and let the query AI deal with the cutover when querying, or do I replay the logs into the new table and accept that the won’t automaticlly be delete in 365 days by the S3 lifecycled rules.