Martin Mueller's Blog

OpenNext + CDK: Next.js on AWS Without ECS Fargate

Sat, 22 Aug 2026 00:00:00 GMT

I usually ship Next.js as Docker standalone on ECS Fargate — ALB, VPC, always-on tasks. For an MVP like qr-plakat.de I went OpenNext 4.x + AWS CDK instead. Cheaper to run, faster to iterate. If the product works, I might switch to Fargate.

Three things made this feel better than the Fargate path. Deploys are one GitHub Actions job from lint to CloudFront invalidation. Infra is CDK I already know — no SST, no extra deploy product. And I can ship from Telegram → OpenClaw → Cursor while watching Netflix or training outside: a PR lands, Actions deploys, I check prod from the phone.

The product: qr-plakat.de

qr-plakat.de is a DACH SaaS for posters with QR codes. You design in a wizard, overlay one to eight codes, and export 300 dpi PDF or JPG. Each poster gets hosted scan URLs like qr-plakat.de/p/{slug} that you can change without reprinting. Everything runs in eu-central-1.

The first real poster is the Brasil shop-window plakat — eight products, eight QR codes, print-ready from the editor.

Subscriptions are mocked. I am the only user, so there is no Stripe checkout. If someone wants a higher tier, they request it in the support chat and I set the plan by hand. Real billing comes when I convince the first business to use the product.

Why OpenNext + CDK for an MVP

OpenNext + CDK is the best choice for MVPs — low cost, fast iteration. If the product works, I might switch to ECS Fargate.

Fargate wants a baseline of tasks, a VPC, and an ALB even when almost nobody is hitting the site. OpenNext maps the Next.js app to pay-per-request Lambda, with CloudFront and S3 in front. I still get App Router, RSC, and Server Actions. I do not run next start in a container.

I stayed on CDK, not SST. Explicit stacks, same IaC I already use. I tried SST's OpenNext path years ago in this post. DynamoDB, S3, and Cognito mean no VPC for the data plane.

The ECS Fargate path I used on listings-mcp is the later-stage option: always-on, long jobs, when traffic and product-market fit justify it.

Stack

Next.js 16 App Router
React 19
shadcn/ui
Tailwind 4
@opennextjs/aws → Lambda ARM64 (2048 MB, 60s)
Lambda + CloudFront + S3
AWS CDK stacks: Data → Auth → Web → Observability (+ Cicd OIDC)
cdk-opennext NextjsSite
Cognito ×2 (creators / admins)
ElectroDB on DynamoDB single-table
Route 53 qr-plakat.de

GitHub Actions — the part I love

One job. No artifact hop between build and deploy. The runner lints, typechecks, and tests, then runs next build, open-next build, and cdk synth. After that it assumes an IAM role with OIDC, runs cdk deploy --all, and invalidates CloudFront /*.

PRs to main ship prod. That is intentional. Dependabot runs CI only.

That loop is why Telegram shipping works. The agent opens a PR, Actions deploys, I hit the live site from the phone.

Agentic loop

I already wrote about OpenClaw and what shipped after three months. On this project the loop is the same, just pointed at qr-plakat.

I send a voice note or a short text on Telegram while watching Netflix or training outside. OpenClaw on the VPS hands the work to Cursor in the real repo. Cursor does a git pull, reads AGENTS.md and docs/, implements the change, and opens a PR. Product decisions stay in docs/ and ADRs — not buried in chat.

MCPs I actually used

Cursor talked to a small set of MCPs on this repo. shadcn for component discovery and install. aws-knowledge-mcp for AWS docs while writing CDK. playwright for browser checks. firecrawl when I needed to scrape or research a page. sistrix for DACH SEO. pdf-reader for print and export PDFs.

Skills that helped

The useful skills were the ones I already had: aws-cdk-development and aws-serverless for infra, vercel-react-best-practices, shadcn-ui, and tailwind-patterns for the app, ci-cd-pipeline-builder for Actions, seo-audit and seo-meta for the landing pages, plus playwright-cli, diagnosing-bugs, and code-review.

There is no OpenNext-specific skill. CDK and serverless skills were enough.

Would I do it again

For an MVP: OpenNext + CDK + one Actions job + Telegram agents. Cheap to run, fast to ship. If qr-plakat takes off, I might move the Next.js app to ECS Fargate. Until then, stay serverless.

Live: qr-plakat.de.

office@martinmueller.dev · calendly.com/martinmueller_dev · LinkedIn

Sovereign Cloud on AWS: When eu-central-1 Is Enough — and When You Need ESC

Fri, 17 Jul 2026 00:00:00 GMT

The AWS European Sovereign Cloud (ESC) has been generally available since January 2026 — physically isolated EU infrastructure, its own partition, EU-resident operations. Across DACH, digital sovereignty is the conversation: data control, operational independence, limiting third-country access.

The core question:

Is eu-central-1 (EU region) enough, or do you need the sovereign partition aws-eusc?

This post is a builder's guide — not AWS marketing. I'm AWS Solutions Architect Professional certified, an AWS Community Builder, and help with ESC migrations (CDK/Terraform), sovereignty decisions, and readiness checks. Resume

Why now?

ESC GA — sovereign AWS workloads are plannable, not "someday"
Region eusc-de-east-1 (Brandenburg) — EU infrastructure with its own governance
Public reference customers: SCHUFA, ITZBund, SAP, EWE, Swiss Life — organizations treating sovereignty as an architecture decision
Politics & procurement: "sovereign cloud", EU-only operations, no third-country access — regardless of which compliance framework applies

There is still little practical builder content — most decks sell sovereignty instead of explaining what breaks in migration.

What ESC actually is — beyond "servers in the EU"

EU region ≠ sovereign partition. Short and factual:

Aspect	Standard AWS (eu-central-1)	AWS ESC (aws-eusc)
Partition	`aws`	`aws-eusc`
Physical isolation	EU region	Dedicated EU infra, separate partition
Operations	Global AWS ops	EU-resident, European governance
Data residency	Yes (EU)	Yes (EU) + isolated control and ops layer
Service scope	Full	~90 services at GA — growing
Typical buyer	Most workloads	Sovereignty-critical data & public sector

eu-central-1 delivers data residency. ESC adds partition isolation and EU-governed operations — that is the sovereignty step, not another certificate.

Decision framework: three buckets

First clarify: what sovereignty requirement do you have? — not which vendor shouts loudest.

Bucket 1 — eu-central-1 is enough

Workloads where EU data residency suffices
No contractual need for isolated partition or EU-only operations
You want the full AWS service catalog without a partition split
Most AWS customers land here — and that's fine

Bucket 2 — ESC for defined workloads

Contractual or political: sovereign partition, no third-country access
Highly sensitive data: health, credit, federal IT, critical infrastructure
Hybrid: core on ESC, rest on eu-central-1 — phased, not big bang

Bucket 3 — Full sovereign stack

ESC alone is not enough politically or legally
Combined with on-prem, BRZ PaaS (AT), Hetzner, or dedicated hosting
Deliberate architecture decision — not the default

In practice: bucket 1 or 2. Sovereignty is workload-specific, not org-wide.

What changes for builders

Marketing skips operational differences between partitions:

Partition & identity

Separate IAM identities — no 1:1 copy from aws
ARN formats and account layout (new landing zone)

DNS & networking

Route 53 behavior and zones — common pain point
Plan or avoid cross-partition references

IaC / CDK

Region eusc-de-east-1, partition aws-eusc
Hardcoded eu-central-1 ARNs break
CI/CD: separate credentials, possibly separate pipelines

Service availability

Serverless stack at GA — verify before design
"Available in eu-central-1" ≠ "available in ESC"

Typical path: readiness → landing zone → pilot → phased migration.

Case studies: who is betting on ESC?

AWS mostly publishes quotes and direction for launch customers — rarely CDK migration paths or account layouts. Still worth studying: why organizations choose ESC and what is publicly known about concrete use.

Common pattern: not org-wide big bang, but sovereignty-critical workloads — often hybrid alongside eu-central-1.

SCHUFA — finance & credit data

Sector: Financial data / credit bureau
Why ESC: 69 million consumer records; isolated EU infra, EU-only staff, no access from outside Europe — sovereignty without sacrificing innovation.
What is known:

Migration from on-premises and other cloud workloads to ESC
New credit scoring system with transparent consumer-facing criteria and weighting
CTO Klaus Kolitz: innovation and sovereignty together, not either/or

Sources: About Amazon — Who's choosing ESC · AWS ESC Customers

Diehl Metering — smart metering & critical infrastructure

Sector: Smart metering (water/energy), B2B2G
Why ESC: Public-sector customers (utilities, waterworks) require EU data residency and EU operations.
What is known:

Smart metering data platform on ESC
Modular services: monitoring, billing
Central system for water/energy data of critical infrastructure operators
Platform scale without building a fully custom sovereign stack

Sources: About Amazon — Who's choosing ESC · The Register (May 2026)

University Hospital Essen — sovereign AI in medicine

Sector: University medicine / research
Why ESC: Health data at scale under German and European sovereignty expectations.
What is known:

IKIM (Institute for AI in Medicine) uses ESC as foundation for AI research
Goal: translate medical research into clinical practice with patient data in a sovereign environment
More transformation and research path than documented full hospital ERP migration

Sources: About Amazon — Who's choosing ESC · AWS Public Sector — Healthcare & Sovereignty

Medizinische Universität Lausitz – Carl Thiem — regional health ecosystem

Sector: Healthcare / research (Brandenburg)
Why ESC: Sovereign digital foundation for the "healthcare model region Lusatia".
What is known:

Connecting hospitals, research, regional partners
Secure sovereign base for care, research, education
Proximity to ESC region Brandenburg (eusc-de-east-1)

Source: AWS ESC Customers

ITZBund — federal IT

Sector: Public sector
Why ESC: Central federal IT provider — strictest protection and sovereignty requirements.
What is known: Commitment to ESC with full AWS service catalog; no public workload list (expected for federal IT).

Source: AWS ESC Customers

EWE AG — energy

Sector: Energy / utilities
Why ESC: Sensitive utility data and critical infrastructure; sovereignty as platform strategy.
What is known: ESC as part of platform strategy — no single named product, but clear strategic bet.

Source: AWS ESC Customers · AWS Launch Press Release

Eterno Health — health SaaS (hybrid thinking)

Sector: Outpatient care / practice software
Why ESC: Digitizing outpatient care across Europe — customers with explicit sovereignty requirements.
What is known:

Full stack on standard AWS today; ESC as additional option for sovereignty-sensitive deployments
Leni — AI agent (transcription, summaries, patient records) tested in own clinics
Pattern: SaaS on AWS, ESC partition for customers who need it

Sources: AWS ESC Customers · AWS Pioneers: ETERNO

Platform partners (not end customers, but relevant)

Vendor	On ESC	Sovereignty angle
SAP	SAP Cloud ERP Private GA	Mission-critical ERP under EU governance
Dedalus	Hospital software (540M+ patients)	Clinical workflows with residency
Arvato Systems	Health Cloud	Sovereign healthcare digitalization

Sources: About Amazon — Who's choosing ESC · AWS Healthcare Sovereignty Blog

What is publicly missing — and where readiness assessments start: IAM/landing zone design, CDK migration paths, timeline, cost. The gap between marketing quote and production migration is large.

ESC vs EU hosting on Hetzner

I wrote about production on Hetzner for EU clients — Arc Rider Universe with full data control on EU VMs.

	Hetzner / self-hosted EU	AWS ESC
Sovereignty model	You operate everything	AWS-native sovereign partition
Audience	New product, full control	Existing AWS shop
Ops burden	High	Lower (managed)
Typical path	SaaS with EU prod requirement	Migrating sovereignty-critical AWS workloads

Both can coexist — hybrid is normal. Sovereignty is not either/or.

CLOUD Act & third-country access — honestly

This is the sovereignty elephant in the room. eu-central-1 hosts in the EU, but the partition stays aws. ESC addresses that directly:

Physically isolated infrastructure in the EU
EU-resident operations — no global US ops access path
European governance structure

What ESC does not replace:

Contract review (subprocessors, support paths)
Data classification — which workloads need which sovereignty level
Political expectations vs technical reality

Honest framing beats marketing. Sovereignty is architecture and contract.

Next step: ESC readiness assessment

If you use or plan AWS and need to clarify sovereignty:

45 minutes, no pitch:

Which bucket (1/2/3) fits your workloads
CDK/Terraform pitfalls (IAM, Route 53, partition)
Migration path — pilot → rollout, hybrid where it makes sense

Talk at AWS Community Day DACH 2026 (Berlin, 15 Sep) — AWS European Sovereign Cloud: A Builder's Guide.

Contact: office@martinmueller.dev · calendly.com/martinmueller_dev · LinkedIn · Resume

Subject: ESC Readiness.

OpenClaw, Three Months Later: What Actually Shipped

Tue, 07 Jul 2026 00:00:00 GMT

In April I wrote about OpenClaw — my self-hosted AI agent on a Hostinger VPS, wired to Telegram, email, calendar, and GitHub. That post ended with a short "What's Next" list. Three months later, most of it shipped — and the setup feels less like a chatbot and more like a small ops team.

What's new since April:

Blog posts — voice note in, researched draft out, published on my site
Code from GitHub issues — agent plans, I approve, PR lands
Invoices — bilingual PDFs for US and German clients, email draft ready to send
SEO & compliance work — live data pulls and reports without me clicking through five tools
One Telegram chat, many projects — separate topics so context doesn't mix
Conference proposals — talk drafts from memory, not from a blank page

Below: what changed, what broke, and what I'd do again.

Quick recap

OpenClaw is a persistent assistant with memory. I talk to it on Telegram — often by voice. It can run tasks, read my repos, draft email, and check GitHub. It only sends mail after I say yes.

Everything below builds on that.

1. Blog posts — from idea to publish

The April post promised blog drafting. That happened — repeatedly:

Post	What the agent did
Hetzner EU production series	Researched the stack, drafted EN+DE, coordinated images
Lovable → Hetzner DACH	Same pattern, tied to a real client story
SISTRIX MCP SEO audit	Pulled live SEO data, read the client's site code, exported a strategy PDF

Typical flow: voice note → agent gathers context from past notes and repos → markdown draft → I edit for tone and accuracy → publish.

Lesson: Great at first drafts with real data. My job is "would I post this under my name?"

2. GitHub issues → plans → PRs

The biggest upgrade: OpenClaw can hand off coding work to Cursor in the actual repo.

Today:

New issue on a client's product repo
I get a Telegram ping
Agent writes an implementation plan from the codebase
Plan lands as a GitHub comment — visible to the team, not buried in chat
I approve → code gets written → PR opens

I stopped burning the expensive model on every small check. Quick triage stays cheap; planning and coding use the heavy models only when it matters.

Lesson: Keep planning in the repo and in GitHub. Telegram is for pings and decisions, not for hiding the plan.

3. Invoices — the boring work I avoided

Invoice day used to mean copy-paste HTML, hunt for rates, second-guess tax wording, export PDF, upload, draft the cover email. Now I open the right Telegram topic and say prepare the next invoice.

The agent already knows each client's rules — US vs German format, currency, VAT, layout. It pulls the last invoice as a template, bumps the number, fills in the period and line items, and asks me only what's missing. Output: PDF in Dropbox, email draft waiting — send only after I approve.

Lesson: Highest ROI automation I didn't mention in April. Repetitive, rule-heavy, easy to get wrong. Perfect for an agent with memory.

4. SEO, WordPress, compliance — one prompt, one artifact

April: basic memory and a few tools. July: the agent can talk to external services I use every week — SEO data, a client's WordPress blog, security scan reports.

The pattern is always the same: one Telegram message, agent combines live data + code context, I get a dated deliverable (markdown, PDF, or issue comment). I wrote up the SEO workflow here.

Lesson: One integration per job you do manually more than once a month. Don't wire everything on day one.

5. Telegram topics = project switcher

One group chat, many topics — blog, US client, German client, side projects.

Each topic loads the right context: which repo, which tone, which rules. I don't mix invoice talk with blog drafting in one thread.

Lesson: If your bot goes quiet in a group, check you're on a recent OpenClaw build. Forum topics had a reply bug that's fixed now.

6. Conference talk drafts

Between client work I still submit to AWS Community Days, AgentCon, CodeMotion, re:Invent. The agent keeps proposal drafts in a folder and knows the submission format.

Typical flow: draft a talk on security scanning for Poland Community Day → agent pulls from past project work and prior talks → title, abstract, outline ready to paste into the submission form → calendar reminder for the deadline.

Lesson: CFP writing is research + structure + your voice. The agent does the first two; I keep the third.

Numbers (rough, three months in)

GitHub: dozens of issues triaged; several fixes shipped via agent → Cursor → PR
Blog: 4+ substantial posts with agent help (EN+DE pairs)
Invoices: 5+ US sprint invoices + German client invoices — PDF + email draft each time
CFP: 10+ proposal drafts across Poland, DACH, Milan, re:Invent
Cost: picking the right model for the task matters more than VPS hosting
Failures: one email sent too early → hard "ask before send" rule; morning briefing broke once after an update — fixed with a version bump and a cheaper scheduled job

What's next

Auto-implement after explicit GitHub approval
Infra changes from chat (Hetzner work in progress)
Quarterly VAT prep from the invoice folder — agent-assisted, accountant still reviews

Conclusion

Three months ago OpenClaw was email, calendar, and GitHub pings. Now it's an orchestrator: topics for context, Cursor for code, live integrations for data, the blog for content, invoices and talk drafts for the business side.

Still on €10/mo Hostinger Docker.

If you already run OpenClaw, what helped me most: hand off coding to Cursor, one Telegram topic per client/project, and one external integration per repetitive manual job.

Links:

Questions or want help wiring your agent? office@martinmueller.dev or calendly.com/martinmueller_dev.

Data-Driven SEO with the SISTRIX MCP: A HalloCasa Case Study

Tue, 23 Jun 2026 00:00:00 GMT

I co-founded HalloCasa — a global real estate broker directory with listings, profiles, subscriptions, and courses. Organic search should be a major channel for us. But when I sat down to build an SEO strategy, I did not want to spend a day clicking through dashboards, exporting CSVs, and stitching screenshots into a doc.

Instead I paired my AI Cursor with the SISTRIX MCP and ran a full audit in one agent session: multi-market visibility baseline, ranking URLs, keyword opportunities, competitor overlap, backlink profile — cross-referenced against our Next.js codebase for canonical tags, hreflang, and sitemaps. The output was a prioritized roadmap, not gut feel.

The whole thing started with one prompt:

Use the SISTRIX MCP to develop a sophisticated SEO strategy for https://hallocasa.com/

No keyword spreadsheets, no elaborate system prompt — just the domain and the tool. From there the agent pulled SISTRIX data market by market, read our repo, and turned it into a strategy doc with prioritized tiers.

This post is about that workflow: what SISTRIX and its MCP are, why MCP matters for SEO, and what the data actually told us about hallocasa.com.

What is SISTRIX?

SISTRIX is a German SEO platform widely used in the DACH market and beyond. Its headline metric is the Visibility Index (Sichtbarkeitsindex) — a score derived from ranking positions and search volume across all keywords a domain ranks for. You also get keyword rankings, backlink data, competitor overlap, search intent, and (newer) AI visibility tooling.

For a site like HalloCasa — global broker directory, LatAm-heavy country list, German expat content on the blog — SISTRIX is useful because you can slice by country (de, us, co, es, mx, …) and compare markets in one session instead of assuming one Google index fits all.

Why MCP for SEO tools?

The Model Context Protocol (MCP) is an open standard that lets AI assistants call external tools through a unified interface. Without it, chatbots guess or hallucinate metrics. With it, they pull live data from your SEO platform.

Without MCP	With MCP
Tab-hopping, manual exports	Agent queries live data in conversation
Stale screenshots in strategy docs	Reproducible, dated API pulls
SEO analyst and developer in separate tools	Agent cross-references rankings and your repo

In Cursor → Settings → Tools & MCP, you wire the SISTRIX server once. The agent then calls tools like domain_visindex or keyword_domain_seo as part of normal chat — same as reading files or running terminal commands.

Workflow (Cursor + SISTRIX MCP + codebase):

Step	You	Agent
1	Ask for SEO audit of your domain	Calls SISTRIX MCP tools per market
2	Point agent at your repo	Reads `seo.ts`, `MetaData.tsx`, `robots.txt`, sitemaps
3	Ask for strategy + priorities	Synthesizes data + code findings into actionable doc

Setting up the SISTRIX MCP

SISTRIX ships an official MCP server. Docs and setup guides:

Resource	Link
Main hub	Connection to AI with MCP
OAuth for all plans (2025+)	SISTRIX MCP Servers changelog
Technical reference	Technical Information MCP
ChatGPT setup	Setting up ChatGPT
Claude setup	Setting up Claude

Endpoint: https://api.sistrix.com/mcp/

Auth: OAuth now works on all SISTRIX plans — log in, no API key required. API keys still work on Plus+ if you prefer. Per SISTRIX docs, MCP requests currently do not consume API credits (unlike direct API calls). For Cursor, configure the MCP server with the same endpoint plus OAuth or an API key in your MCP config.

SISTRIX MCP tool surface

The MCP exposes dozens of tools. In the HalloCasa audit we leaned on these groups:

Category	Example tools	What you learn
Domain visibility	`domain_visindex`, `domain_kwcount_seo`, `domain_ranking_distribution`	Footprint per market, page-1 keyword count
Keywords	`keyword_domain_seo`, `domain_opportunities`, `keyword_seo_metrics`	What ranks, near-wins, volume and intent
Competition	`domain_competitors_seo`, `domain_ideas`	Overlap and content gaps
Links	`links_list`	Toxic vs legitimate backlinks
AI visibility	`ai_entity`	AEO baseline (our HalloCasa entity call returned HTTP 500)
Projects	`project_ranking`, `project_visibilityindex`, …	Ongoing tracking once a project exists

Pass country as an ISO code (de, us, co, es, mx, …) to audit multiple markets in one session. That is how we discovered Germany has history while Colombia and Mexico have essentially zero visibility today.

HalloCasa case study: what the data showed

Domain: hallocasa.com. The agent pulled SISTRIX data across our main markets (DE, US, ES, CO, MX).

Baseline snapshot

Germany is the only market with meaningful organic history — visibility has declined sharply since a peak a few years ago. The US and Spain show traces of activity (mostly blog content). Colombia and Mexico, despite being core business geography, had essentially no organic footprint at all.

That mismatch between business focus and search visibility was the first big insight — and only visible because we queried per country instead of treating Google as one global index.

Where rankings actually live

Almost all SEO value sits on blog.hallocasa.com, mostly German articles about LatAm real estate (Chile, Colombia, Peru, Argentina, Mexico). The directory (www.hallocasa.com) barely ranks by comparison — a handful of broker-name profile queries and little else.

Headline finding: authority is stranded on a subdomain, not flowing to broker profiles and /brokers. That is a structural problem, not a content gap.

Near-win keywords (fastest ranking upside)

SISTRIX domain_opportunities surfaced several German keywords where we already rank on page 2 — cross-border investment and expat real estate topics with relatively low competition. Pushing those over the line to page 1 is lower effort than chasing new head terms.

Competitors (DE organic overlap)

The overlap set is a mix of large German property portals, international luxury listing sites, and niche expat/investment blogs. We are not competing on generic German portal head terms; our wedge is cross-border / expat investor content.

Backlinks: mostly noise

The link profile is dominated by SEO directory and PBN spam. Legitimate links exist but are sparse — partner podcasts, travel sites, conference media. Worth a GSC audit before any disavow; Google often ignores obvious spam automatically.

Code cross-check: why the directory does not rank

The agent read our Next.js repo alongside SISTRIX URLs. Technical issues that explain weak directory visibility:

i18n via ?lang= on the directory, but canonical URLs strip the query — all nine locales collapse to one indexable URL per path.
No hreflang anywhere; blog and home use path locales (/de/, /es/), apex uses query locales.
Fragmented architecture: blog., home., and apex as separate properties; blog holds the rankings.

Fixing international SEO and consolidating blog authority onto the main domain are foundation work before scaling content in CO/MX.

Prioritized roadmap (90-day lens)

From the audit we tiered actions by effort vs impact:

P0 — do first (weeks 1–4)

Action	Why
GSC domain property + baseline KPIs	Cannot measure without it
`www` canonical + 301 all host variants	Stops split link equity
Blog → directory CTAs on top DE posts	Rankings exist; directory gets no traffic
Cross-link blog ↔ `/brokers` ↔ profiles	Partial authority transfer without migration

P1 — highest SEO upside (weeks 4–16)

Action	Why
Optimize near-win DE blog posts	Fastest ranking wins
Blog → `/blog/` subfolder consolidation	Biggest structural lift for directory
Path locales + `hreflang` on apex	Required before ES/EN scale
DE pillar hub for cross-border investment	Captures commercial intent cluster

P3 — long bets

Spanish LatAm programmatic pages (CO/MX) only when broker inventory supports it — empty city templates would be thin-content risk.

How to replicate this workflow

Wire SISTRIX MCP in Cursor — endpoint https://api.sistrix.com/mcp/, setup docs, OAuth or API key.
Open with a clear ask — e.g. Use the SISTRIX MCP to develop a sophisticated SEO strategy for https://hallocasa.com/ (that was my entire starting prompt).
Ask for multi-market baseline — domain_visindex and domain_kwcount_seo per country.
Pull ranking URLs — keyword_domain_seo to see which subdomain or path actually ranks.
Pull opportunities, competitors, backlinks — domain_opportunities, domain_competitors_seo, links_list.
Audit your codebase — canonicals, hreflang, robots, sitemaps, i18n patterns.
Synthesize — strategy doc with prioritized tiers, not a data dump.

Sample deliverable: HalloCasa SEO strategy (PDF)

The agent session produced a full strategy document — baseline by market, competitor overlap, technical findings from our Next.js repo, and a P0–P3 roadmap. We exported it as PDF for stakeholders (pandoc + headless Chrome from the markdown in hallocasa-next/misc/seo).

Download PDF · View markdown source

Want something similar for your domain? I help teams run the same SISTRIX MCP + codebase audit workflow and turn it into a prioritized strategy doc like this. Reach out on LinkedIn.

Takeaways for other sites

MCP turns SEO from dashboard archaeology into agent-native research. The agent pulls live SISTRIX data while you stay in the editor.
Best results when the agent can read your repo too. Rankings tell you what ranks; code tells you why fixes are hard or easy.
Multi-subdomain setups need an explicit architecture decision — consolidate (e.g. blog → /blog/) or cross-link heavily; do not assume authority transfers by default.
Prioritize near-win keywords before greenfield programmatic SEO or nine-locale translation sprees.
Match market data to business focus. We had almost no visibility in CO/MX despite LatAm being core geography — that mismatch only shows up when you query per country.

What is next for HalloCasa

This post is about the research method — but the audit gave us a clear roadmap, and I am excited to implement it: P0 fixes first, then the DE near-win posts and blog consolidation. SISTRIX will stay in the loop for ongoing tracking, not just this one-off session.

The same MCP setup opens GEO (Generative Engine Optimization): SISTRIX’s AI visibility tools (ai_entity, and related project tooling) let us baseline how HalloCasa shows up in AI answers and measure lift as we fix structure and content. Our first ai_entity call hit a 500 — worth retrying once the foundation work lands.

I am co-founder of HalloCasa and use MCP heavily elsewhere in my stack (see How I Use OpenClaw for another angle on agent tooling). If you want help wiring SISTRIX MCP in Cursor or running a similar audit on your domain, reach out on LinkedIn.

Thanks to Benedikt Rudolf at SISTRIX for taking the time to walk me through the MCP — that conversation kicked off this workflow.

Built with Lovable? Host Prod in Germany & DACH on Hetzner

Mon, 08 Jun 2026 00:00:00 GMT

You built on Lovable — React app, Supabase auth, Postgres. It works. Then your first DACH customer or pilot asks: Where is prod hosted? Where do login and data live? That question shows up fast in Germany, Austria, and Switzerland B2B — especially HR, health, finance, and anything that touches procurement or a security questionnaire.

Why “EU is fine” is not enough in DACH

Lovable’s default path — Netlify (or similar) plus managed Supabase — is great for building. It is not always enough for prod you can point to in a buyer review.

Germany — GDPR, vendor questionnaires, and “EU hosting” often means nameable EU infrastructure, not “we use a US cloud somewhere in eu-west-1.”
Austria — Same GDPR frame; many buyers mirror German expectations on data location and subprocessors.
Switzerland — Not in the EU/GDPR, but strong privacy expectations (revDSG). Hetzner — German company, EU datacenters — is a credible answer for many Swiss buyers without overclaiming compliance.

This is not legal advice. It is the practical friction you hit when DACH customers want a clear answer on where prod runs.

Phased prod on Hetzner — dev stays on Lovable

You do not need a big-bang migration. In a recent German-client case study, dev stayed on Netlify + managed Supabase while prod — app, login, Postgres — went live on Hetzner in the EU. Same pattern fits Lovable stacks: keep building in Lovable; add a DACH-facing prod environment on EU VMs.

Case study:

Production on Hetzner (Part 1) — phased EU prod, OpenTofu, CI deploys, self-hosted Supabase
Self-hosted insights (Part 2) — observability, cost guardrails, EU telemetry

Prod users hit the React app on Hetzner. Sign-in and Postgres for prod stay on Hetzner too. Dev keeps using Lovable’s stack unchanged.

Open source stack — no black box prod

DACH buyers often ask not only where data lives but what software runs there. The prod path we use is built on open source you can name, audit, and run yourself — not a proprietary PaaS you cannot inspect.

Layer	Open source
Infra as code	OpenTofu (an open-source fork of Terraform)
App runtime	React (from Lovable), nginx
Auth + API + DB	Supabase (self-hosted), Postgres
Observability	Grafana, Prometheus, Loki, Tempo, OpenTelemetry

That matters in security reviews: subprocessors and components are identifiable. You are not locked into a single vendor’s managed layer for prod. Hetzner gives you VMs; the stack on top stays portable and forkable.

What I handle vs what you keep doing

You: keep shipping features in Lovable.

Me: OpenTofu (servers, firewall, DNS), GitHub Actions deploys, self-hosted open-source Supabase on prod VMs, and self-hosted observability (Grafana LGTM, cost alerts) — the documentation DACH buyers often ask for: where data lives, what runs there, who operates prod, what it costs.

I work with DACH clients on exactly this path. If that sounds like your next step, reach out.

office@martinmueller.dev · calendly.com/martinmueller_dev · LinkedIn

Production on Hetzner for a German Client — EU Hosting (Part 1)

Fri, 05 Jun 2026 00:00:00 GMT

Part 2: Self-hosted insights on Hetzner

A German client needed production in the EU on Hetzner — clear control over where customer data and login live. The product is Arc Rider Universe: the SaaS behind universe.arc-rider.com — Mission: Interface, a UI toolkit for builders (tables, Kanban, Gantt, layouts, and more via JSON instead of hand-rolled UI code). Dev already ran happily on Netlify and managed Supabase — and still does. The goal was a dedicated prod environment on Hetzner, not a big-bang migration that would slow down the team.

Why Hetzner, and why not “move everything at once”

The client is in Germany. Prod had to sit on EU infrastructure they can point to — Hetzner VMs for the app and the database.

That did not mean touching dev:

dev.universe.arc-rider.com (dev) stays on Netlify + managed Supabase.
Prod — app, login, Postgres — runs on Hetzner at universe.arc-rider.com.

If you are planning something similar: you can add EU prod on Hetzner in one weekend — without replatforming dev.

What prod looks like (in plain terms)

Prod users open the React app on Hetzner. Sign-in and data for prod live on Hetzner too. Dev keeps using Netlify and managed Supabase unchanged.

Infra and deploys are automated with OpenTofu (servers, firewall, DNS) and GitHub Actions (build, deploy, quick smoke checks after a release).

Once prod was stable, we added self-hosted observability on a dedicated VM — metrics, logs, traces, cost guardrails, and agent-friendly ops. See Part 2.

What we learned (short version)

Phased beats big bang — dev stayed on Netlify + managed Supabase while Hetzner prod came online.
Self-hosted login is not a copy-paste from cloud Supabase — prod OAuth and magic links need prod-specific URLs and config; dev config stays separate.
Predictable cost and EU hosting — VMs on Hetzner for app, login, and data.

Prod is live at universe.arc-rider.com. For this product stage, the tradeoff worked: control where it mattered on EU infrastructure the client can point to.

Hetzner migrations — how I can help

On Netlify, managed Supabase, or AWS for dev and need prod in the EU on Hetzner? I help with planning, OpenTofu, CI deploys, self-hosted Supabase on prod VMs, and self-hosted observability (Grafana LGTM, Alloy, cost guardrails, Grafana MCP for agent-friendly ops).

Built with Lovable for Germany/DACH? Same phased EU-prod path.

office@martinmueller.dev · calendly.com/martinmueller_dev · LinkedIn

Self-Hosted Insights on Hetzner — Observability, Cost, AI Agents (Part 2)

Fri, 05 Jun 2026 00:00:00 GMT

Part 1: EU production on Hetzner

After prod went live on Hetzner — React app, self-hosted Supabase login, Postgres on EU VMs; dev still on Netlify + managed Supabase — we added self-hosted observability on a dedicated prod VM. Metrics, logs, traces, and cost data stay in the EU. No third-party APM SaaS. Then we wired AI agents to query prod through Grafana MCP instead of SSH guesswork.

Insights without leaving the EU

We added self-hosted observability on Hetzner: OpenTelemetry ingest, Grafana LGTM (Prometheus, Loki, Tempo), and Grafana Alloy agents on the static and Supabase VMs. Browser RUM and edge-function traces and logs export to the same stack — nothing leaves the EU for “how is prod doing?”

Provisioned dashboards (git → redeploy):

Dashboard	What it shows
Data health	Are metrics, logs, and traces flowing?
Prod Overview	High-level prod health
Infrastructure USE	CPU, memory, disk on VMs
nginx RED	Request rate, errors, duration from JSON access logs
Supabase Docker	Container health on the API VM
Postgres	Connections, query stats
Frontend RUM	Browser traces (Tempo)
Edge Functions	Edge traces + app logs in Loki

Why this matters for clients: you get Datadog-class visibility without shipping telemetry to US SaaS or guessing from SSH. Dashboards and alert rules live in git; Google OAuth gates the Grafana UI.

Predictable cost, watched automatically

Hetzner pricing is transparent — but three VMs, backups, primary IPs, and volumes still add up. We wired a Production Costs dashboard that pulls live Hetzner API pricing into Prometheus (arc_infra_cost_eur_monthly): per-server, backup, volume, and primary IP breakdown, plus total monthly and annual projection.

A Grafana alert fires when estimated infra cost hits EUR 50/month — email to the ops list before the invoice surprises anyone. That is the opposite of opaque cloud bills where you discover the spike in the console.

AI agents with real tools

A lot of this was designed, debugged, and operated with AI assistants in the loop (Cursor, Claude Code). The standout learning: once Grafana is self-hosted on Hetzner, you can point agents at prod itself — not just docs and IaC.

With Grafana MCP (mcp-grafana via uvx) and a read-only Viewer service-account token, an agent can:

Run PromQL (host CPU, Postgres connections, cost metrics)
Run LogQL (nginx access logs, edge-function output)
Search dashboards and summarize panels
List alert rules and check firing state

That turned “something is slow in prod” into a loop: agent queries Grafana → finds the panel or log stream → suggests or applies a fix in git/CI — without SSH guesswork. Tool approval stays on in Cursor; the token is read-only; infra and schema changes still go through OpenTofu and CI.

In Cursor → Settings → Tools & MCP, the project wires MCP servers plus the usual CLIs:

Tool	What we used it for
hcloud CLI	Check servers, IPs, firewalls from the terminal
OpenTofu CLI	Plan and apply infra (VMs, firewall, DNS)
OpenTofu MCP (`@opentofu/opentofu-mcp-server` via `npx`)	Look up provider docs in chat (read-only)
Supabase CLI	Migrations and schema work in git
Supabase MCP (official hosted MCP)	Inspect tables, run read-only SQL, check logs and docs
Grafana MCP (mcp-grafana via `uvx`)	PromQL, LogQL, dashboards, alerts on self-hosted Grafana

{
 "mcpServers": {
  "supabase": {
   "url": "https://mcp.supabase.com/mcp?project_ref=YOUR_REF&read_only=true"
  },
  "opentofu": {
   "command": "npx",
   "args": ["-y", "@opentofu/opentofu-mcp-server"]
  },
  "grafana": {
   "command": "uvx",
   "args": [
    "mcp-grafana",
    "--disable-admin",
    "--disable-oncall",
    "--disable-incident"
   ],
   "env": {
    "GRAFANA_URL": "https://grafana.prod.universe.arc-rider.com",
    "GRAFANA_SERVICE_ACCOUNT_TOKEN": "YOUR_GRAFANA_SERVICE_ACCOUNT_TOKEN"
   }
  }
 }
}

Together with runbooks, that felt closer to IaC plus observability plus CLIs than to “SSH into a pet server and hope.” Applies still go through OpenTofu/CI; MCP is for lookup, schema, logs, and prod queries — not silent infra or schema writes.

What we learned (short version)

Observability on Hetzner is a product asset — metrics, logs, and traces in the EU, provisioned from git, no APM SaaS lock-in.
Predictable cost and EU hosting — cost dashboard + budget alert so spend stays visible before the invoice surprises anyone.
AI agents shine on self-hosted Grafana — Grafana MCP + read-only token lets agents investigate prod (PromQL, LogQL, dashboards) instead of guessing from SSH.

Prod is live at prod.universe.arc-rider.com. For this product stage, the tradeoff worked: control where it mattered on EU infrastructure the client can point to — including how prod is watched and what it costs.

Hetzner migrations — how I can help

Built with Lovable for Germany/DACH? Same phased EU-prod path.

office@martinmueller.dev · calendly.com/martinmueller_dev · LinkedIn

How I Use OpenClaw as My AI-Powered Personal Operating System

Wed, 01 Apr 2026 00:00:00 GMT

I've been running OpenClaw on a Hostinger VPS for about a month now, and it has fundamentally changed how I work. What started as "let me try this AI agent thing" turned into a powerful assistant that saves me hours every week by automating routine tasks — from GitHub issue triage to email management.

OpenClaw doesn't do everything for me (yet), but it handles a lot of the repetitive work that used to eat up my day. Here's a deep dive into every use case I've discovered so far.

What is OpenClaw?

OpenClaw is an open-source AI agent platform that runs on your own infrastructure. You connect it to your chat channels (Telegram, WhatsApp, Discord), give it access to your tools, and it becomes a persistent assistant that remembers context across sessions. Think of it as your own self-hosted AI employee that's always on.

My setup: Docker container on a Hostinger VPS, with Telegram as my primary interface. Most of my interactions — voice messages, text, files — go through Telegram. It feels like texting a colleague who never sleeps.

Use Case 1: Autonomous GitHub Issue Management

This is the killer feature for me. OpenClaw monitors my HalloCasa repositories every 2 hours via a heartbeat. When a new issue appears, it:

Notifies me on Telegram
Generates a detailed implementation plan using Cursor CLI with Claude Opus for planning
Posts the plan as a comment on the GitHub issue
Waits for my approval before implementing
Implements the fix using Cursor CLI with Composer
Creates a branch, commits, pushes, and opens a PR

In the first week alone, it planned and implemented fixes for currency display bugs, locale changes, phone validation, and filter cleanup — all with minimal intervention from me.

The workflow follows the "Factory" pattern I heard about at Agentic Conf Hamburg: don't just use AI to build features — build a production line that builds features for you.

Use Case 2: Email Management

OpenClaw reads and sends emails via Himalaya CLI (IMAP/SMTP). I just tell it what to do on Telegram:

"Tell the sports group leader we can't come today" → Searched my contacts, found the match, sent a polite cancellation.
"Email my tax advisor that I need to reschedule" → Found the firm in contacts, sent a professional German email.
"Reply to the kindergarten about scheduling a visit" → Continued an existing email thread with context-aware reply.

It sends HTML emails with a professional signature and handles German and English.

Use Case 3: Morning Briefing (Cron Job)

Every morning at 7:00 AM (my timezone, Europe/Berlin), OpenClaw runs an automated check:

Scans unread emails from the last 24 hours
Checks yesterday's emails for pending follow-ups
Cross-references with my Google Calendar for today and tomorrow
Sends me a concise summary on Telegram

This runs on Claude Haiku (cheap) and costs almost nothing. I wake up to a briefing instead of manually checking three different apps.

Use Case 4: Calendar Management

OpenClaw has full access to my Google Calendar via OAuth. Examples:

Conference schedule: I sent a PDF of the Agentic Conf Hamburg schedule with sessions I'd marked with red boxes. It extracted the image, identified the marked sessions using vision AI, fetched details from the conference website, and created 9 calendar events with full descriptions, speaker info, and links.
Creating meetings: "Create a meeting with Phillip P. for today at 18:00 CET" → Done.

Use Case 5: Contact Lookup

Connected to my iCloud contacts via CardDAV, OpenClaw searches by name, organization, or context. When I say "email the sports group leader," it finds the match and uses the right email. No manual lookup needed.

Use Case 6: LinkedIn Post Drafting

I've used it to draft LinkedIn posts for:

AWS Community Day Athens 2026: Looked up all speakers from the conference website, compiled them alphabetically, created a professional announcement post.
Agentic Conf Hamburg recap: Searched for organizers' backgrounds, found LinkedIn profiles, referenced specific talks, and wove in my personal highlights about the "Factory" concept.

It does the research, I do the personal touch.

Use Case 7: Research & Due Diligence

Questions that would normally cost 15 minutes of Googling:

"Is there an app for biometric passport photos with QR codes?" → Comparison of available apps, including recent regulation changes.
"There's a conference in my area tomorrow, what is it?" → Found the event, full program, speakers, venue, and registration link.

Use Case 8: Document Generation

Birthday invitation: Generated a themed HTML invitation card with embedded images and QR code, published as a shareable link.
Application form: Built a bilingual job application form, deployed to GitHub Pages so it works when shared via WhatsApp.
Kindergarten applications: Generated PDF application forms from templates.
Email signatures: Set up HTML signatures in Gmail (via API) and Apple Mail.

Use Case 9: Workspace & Config Management

OpenClaw manages its own configuration:

Version-controls its workspace files in a Git repo
Adjusts its own heartbeat intervals and model settings when I ask
Maintains daily memory notes and long-term memory files for context continuity

Use Case 10: Global Brain via PeachBase

OpenClaw is connected to PeachBase — a serverless vector database that acts as my persistent memory across all AI agents. Via MCP, OpenClaw can store and retrieve knowledge: personal info, project decisions, contacts, learnings.

When I tell OpenClaw something worth remembering, it stores it in PeachBase. When I ask a question weeks later — even from a different agent like Cursor — the knowledge is there. It's the shared brain that ties everything together.

I wrote a dedicated post about this: My Global Brain with PeachBase.

Use Case 11: Multi-Channel Communication

I talk to OpenClaw primarily via Telegram, but it also:

Sends emails on my behalf (Himalaya/SMTP)
Is connected to WhatsApp
Can deliver cron job results to specific Telegram chats

Use Case 12: Writing This Blog Post

Meta moment: this very blog post was drafted by OpenClaw. I sent a voice message on Telegram (in German): "I want to write about OpenClaw and how I've been using it — go through all your history and find the use cases, in English please."

It scanned 23 days of daily memory notes, extracted every use case, researched the blog repo format, and produced a full draft in the right Gatsby frontmatter format — all in one turn. I just reviewed and tweaked.

This is the "Factory" idea in action: I didn't write a blog post. I told my agent to write one, and it had all the context it needed because it was there for every use case.

Security: It's a Spectrum

Giving an AI agent access to your email, contacts, calendar, and GitHub is powerful — but it's also a security conversation you need to have with yourself.

OpenClaw treats security as a spectrum, not a binary. You can start wide-open for convenience and tighten as you go. Here's what's available and what I use:

Sandboxing

OpenClaw supports Docker-based sandboxing where tool execution (shell commands, file reads/writes) runs inside an isolated container instead of directly on the host. You can choose:

"off" — everything runs on the host (my current setup, maximum convenience)
"non-main" — only non-main sessions (group chats, webhooks) are sandboxed
"all" — every session runs sandboxed

I'm still on "off" because I'm the only user and I trust the agent boundary. But if you're running OpenClaw on a shared machine or exposing it to group chats, sandboxing is a must.

Tool Policy

You can allowlist or denylist specific tools per agent. For example, you could disable exec (shell access) entirely and only allow read/write — or restrict which commands can run. OpenClaw also has an elevated exec model (think sudo) where dangerous commands require explicit approval.

Secrets

One thing I'd do differently: my early setup had API keys in config files. OpenClaw supports environment variables and secret refs instead. Move your credentials out of plaintext config.

The Human-in-the-Loop Rule

My most important security measure isn't technical — it's a rule in my agent's config: always ask before sending emails. After one incident where the agent sent an email I hadn't approved, I added a hard rule. The agent now shows me a draft and waits for "OK" before any outbound communication. This applies to anything that "leaves the machine" — emails, social posts, webhooks.

Network

The Gateway port (18789) should never be public. Mine is localhost-only inside Docker. If you need remote access, use Tailscale or an authenticated reverse proxy with TLS.

My Take

Security with AI agents is genuinely new territory. The threat model is different from traditional apps because the agent can be influenced by external content (prompt injection via emails, web pages). OpenClaw has tools to limit blast radius — sandboxing, tool policies, approval gates — but the most important thing is being deliberate about what you give access to and expanding gradually.

The Numbers

After a month of use, here's what surprised me:

Cost optimization matters: Running on Claude Opus 4.6 burns through API credits fast. Switching heartbeats to Haiku and reducing intervals from 30min to 2h made a huge difference.
Memory is everything: The daily notes + MEMORY.md system means I never have to re-explain context. It knows my projects, my contacts, my preferences.
HTML email was harder than expected: Getting Himalaya to send proper HTML emails with MML syntax took some trial and error. Plain text signatures don't have clickable links.

What's Next

Blog post automation: Using OpenClaw to help draft and publish posts (like this one!)
Deeper GitHub integration: Auto-implementing approved plans without manual trigger
More cron jobs: Weather briefings, social media monitoring, calendar reminders

Conclusion

OpenClaw isn't just a chatbot. It's a personal operating system that happens to be powered by AI. The combination of persistent memory, tool access (email, calendar, contacts, GitHub, file system), and multi-channel communication makes it genuinely useful — not in a "cool demo" way, but in a "I saved 2 hours today" way.

If you're a developer comfortable with Docker and CLI tools, I highly recommend giving it a try. The learning curve is worth it.

Links:

🚀 I'm also looking for beta testers for PeachBase — the serverless vector DB I use as shared memory across all my AI agents. If you want to try it, sign up!

Thanks for reading! If you have questions about my setup, feel free to reach out. And if you'd like help setting up your own OpenClaw AI agent — whether it's configuration, tool integration, or building custom workflows — I'm available for consulting. Just drop me a message at office@martinmueller.dev or book a call at calendly.com/martinmueller_dev.

Strands TypeScript SDK - Building Production AI Agents

Fri, 30 Jan 2026 00:00:00 GMT

Introduction

Building AI agents that work in production requires more than wrapping an LLM API. You need tool execution, streaming responses, cost management, and integration with existing systems. After evaluating several frameworks for ai-secure.dev, I chose the Strands TypeScript SDK from AWS.

Why Strands over alternatives?

Framework	Pros	Cons
LangChain	Feature-rich, large ecosystem	Heavy, complex abstractions
crewAI	Multi-agent orchestration, role-based agents	Python-focused, heavier runtime
Raw Anthropic/OpenAI API	Full control	Too low-level, no tool orchestration
Strands SDK	Lightweight, AWS-native, streaming-first	Newer, smaller community

Strands hits the sweet spot: enough abstraction to be productive, low enough to maintain control. It's what I used to build the security audit agent behind ai-secure.dev.

Agent Creation Basics

Creating an agent requires three things: a model, a system prompt, and tools.

import { Agent, tool } from '@strands-agents/sdk'
import { z } from 'zod'

const agent = new Agent({
  model,                    // BedrockModel or custom provider
  systemPrompt: `You are a security auditor...`,
  tools: [httpSecurityCheck, dnsLookup, browserNavigate],
})

// Invoke the agent
const response = await agent.invoke('Audit https://example.com')

// Or stream for real-time updates
for await (const event of agent.stream(prompt)) {
  // Handle events: text deltas, tool calls, metadata
}

The SDK handles the agentic loop: model generates response → tool calls extracted → tools executed → results fed back → repeat until done.

Defining Tools

Tools are functions the agent can call. The tool() helper wraps them with Zod schema validation:

const calculatorTool = tool({
  name: 'calculator',
  description: 'Performs arithmetic. Params: operation, a, b',
  inputSchema: z.object({
    operation: z.enum(['add', 'subtract', 'multiply', 'divide']),
    a: z.number(),
    b: z.number(),
  }),
  callback: (input) => {
    let result: number
    switch (input.operation) {
      case 'add': result = input.a + input.b; break
      case 'subtract': result = input.a - input.b; break
      // ...
    }
    return `Result: ${result}`
  },
})

For domain-specific agents, design tools around your use case. My security agent has tools like:

http_security_check - Headers, TLS inspection, redirect chain
dns_lookup - SPF/DMARC/CAA records
browser_navigate - Navigate and interact with pages
totp - Generate 2FA codes for authenticated scans

Complex tool example (abbreviated):

const httpSecurityCheckTool = tool({
  name: 'http_security_check',
  description: 'HTTP security analysis: headers, TLS cert, redirects',
  inputSchema: z.object({
    url: z.string().describe('URL to check'),
    method: z.enum(['GET', 'HEAD', 'OPTIONS']).optional(),
    includeTls: z.boolean().optional(),
  }),
  callback: async (input) => {
    // Make request, inspect TLS socket, check headers
    const securityHeaders = ['strict-transport-security', 'content-security-policy', ...]
    // ... implementation
    return JSON.stringify({ url, statusCode, securityHeaders, tls })
  },
})

Tools are the agent's "hands" - design them for your domain, not as generic utilities.

Custom Model Provider

The SDK includes BedrockModel for AWS Bedrock, but you can create custom providers. I built AnthropicModel for direct Anthropic API access with features like message caching:

export class AnthropicModel {
  constructor(config: AnthropicModelConfig) {
    this.client = new Anthropic({ apiKey: config.apiKey })
    this.config = {
      modelId: config.modelId || 'claude-sonnet-4-5-20250929',
      maxTokens: config.maxTokens || 16000,
      enableMessageCaching: config.enableMessageCaching ?? true,
    }
  }

  async *stream(messages, options) {
    // Convert messages to Anthropic format
    // Add cache_control blocks for cost reduction
    // Yield SDK-compatible events
  }
}

Message caching reduces costs by 90% on repeated context. Add cache_control to strategic messages:

// Cache system prompt (reused every call)
request.system = [{
  type: 'text',
  text: systemPrompt,
  cache_control: { type: 'ephemeral', ttl: '1h' }
}]

// Cache last tool definition
tools[tools.length - 1].cache_control = { type: 'ephemeral', ttl: '1h' }

Cost tracking built into the model:

const MODEL_PRICING = {
  'claude-sonnet-4-5-20250929': { input: 3.00, output: 15.00, cacheRead: 0.30 },
  'claude-haiku-4-5-20251001': { input: 1.00, output: 5.00, cacheRead: 0.10 },
}

function calculateCost(modelId, inputTokens, outputTokens, cacheReadTokens) {
  const pricing = MODEL_PRICING[modelId]
  return (inputTokens * pricing.input + outputTokens * pricing.output 
          + cacheReadTokens * pricing.cacheRead) / 1_000_000
}

Model Routing for Cost Optimization

Not every request needs your most powerful model. Route simple tasks to cheaper models:

function classifyTask(prompt: string) {
  const lower = prompt.toLowerCase()
  
  // Complex patterns → Sonnet
  const complexPatterns = [
    /security|vulnerabil|audit/i,
    /iso\s*27001|compliance/i,
    /investigate|analyze|assess/i,
  ]
  
  // Simple patterns → Haiku (10x cheaper)
  const simplePatterns = [
    /^(hi|hello|hey)/i,
    /^(thanks|thank\s*you)/i,
    /^(yes|no|ok)/i,
  ]
  
  for (const pattern of complexPatterns) {
    if (pattern.test(prompt)) {
      return { complexity: 'complex', model: 'claude-sonnet-4-5' }
    }
  }
  
  for (const pattern of simplePatterns) {
    if (pattern.test(lower)) {
      return { complexity: 'simple', model: 'claude-haiku-4-5' }
    }
  }
  
  // URLs always complex (security audits need full power)
  if (prompt.includes('http://') || prompt.includes('https://')) {
    return { complexity: 'complex', model: 'claude-sonnet-4-5' }
  }
  
  return { complexity: 'complex', model: 'claude-sonnet-4-5' } // Default safe
}

Log cost comparisons in production to validate routing:

📊 Tokens: 15420 in, 2341 out | $0.0812 (sonnet-4-5)
   Alternative: $0.4102 (opus-4-5) → +$0.329 (+405%)

Streaming Architecture

For real-time UX, stream agent events via Server-Sent Events (SSE):

app.post('/invocations', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream')
  res.setHeader('Cache-Control', 'no-cache')
  
  const sendEvent = (type, data) => {
    res.write(`data: ${JSON.stringify({ type, ...data })}\n\n`)
  }
  
  for await (const event of agent.stream(prompt)) {
    // Text streaming
    if (event.type === 'modelContentBlockDeltaEvent') {
      const delta = event.delta
      if (delta?.type === 'textDelta') {
        sendEvent('text', { content: delta.text })
      }
    }
    
    // Tool execution tracking
    if (event.type === 'modelContentBlockStartEvent') {
      const start = event.start
      if (start?.type === 'toolUseStart') {
        sendEvent('tool_start', { tool: start.name })
      }
    }
    
    if (event.type === 'afterToolsEvent') {
      sendEvent('tool_end', { tool: currentTool })
    }
    
    // Token usage
    if (event.type === 'modelMetadataEvent') {
      totalTokens += event.usage?.totalTokens || 0
    }
  }
  
  sendEvent('done', { usage: { totalTokens } })
  res.end()
})

Key event types:

Event	When	Use
`modelContentBlockDeltaEvent`	Text/tool input streaming	Real-time display
`modelContentBlockStartEvent`	Tool call begins	Show "Analyzing..."
`afterToolsEvent`	Tool finished	Show result
`modelMetadataEvent`	Tokens counted	Cost tracking

OpenAI-Compatible Adapter

Why build this? During development, I needed to chat with my agent without building a UI first. By exposing an OpenAI-compatible endpoint, I could use Cline (VS Code extension) as my interface - instant chat UI for free.

This let me iterate on tools and prompts rapidly before touching frontend code.

export function createOpenAIAdapter(config) {
  const router = Router()
  
  router.get('/v1/models', (_, res) => {
    res.json({
      data: [{ id: config.modelName, owned_by: 'strands-agents' }]
    })
  })
  
  router.post('/v1/chat/completions', async (req, res) => {
    const { messages, stream } = req.body
    const prompt = extractPromptFromMessages(messages)
    
    const { agent } = config.createAgent()
    
    if (stream) {
      // Stream SSE chunks in OpenAI format
      res.setHeader('Content-Type', 'text/event-stream')
      for await (const event of agent.stream(prompt)) {
        // Convert to OpenAI chunk format
        res.write(`data: ${JSON.stringify(chunk)}\n\n`)
      }
      res.write('data: [DONE]\n\n')
    } else {
      // Collect and return
      const response = await agent.invoke(prompt)
      res.json({ choices: [{ message: { content: response } }] })
    }
  })
  
  return router
}

// Mount the adapter
app.use(createOpenAIAdapter({ modelName: 'security-agent', createAgent }))

Now point Cline at http://localhost:8080/v1 and it works.

Production Tips

Session management with TTL:

const sessions = new Map<string, Session>()
const SESSION_TTL_MS = 30 * 60 * 1000 // 30 min

setInterval(() => {
  const now = Date.now()
  for (const [id, session] of sessions) {
    if (now - session.lastAccessedAt > SESSION_TTL_MS) {
      sessions.delete(id)
    }
  }
}, 60 * 1000)

Issue tracking during scans:

const issueTrackerTool = tool({
  name: 'issue_tracker',
  description: 'Track problems during audit: auth failures, timeouts, etc.',
  inputSchema: z.object({
    type: z.enum(['auth_failed', 'access_denied', 'timeout', 'credentials_required']),
    title: z.string(),
    description: z.string(),
  }),
  callback: (input) => {
    session.issues.push(input)
    return `Issue tracked: ${input.title}`
  },
})

Include issues in the final report so users know what couldn't be tested.

Architecture Overview

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│   Frontend  │────▶│  Agent Server    │────▶│   Tools     │
│   (Next.js) │     │  (Strands SDK)   │     │ (http, dns, │
└─────────────┘     │                  │     │  browser)   │
      ▲             └────────┬─────────┘     └─────────────┘
      │                      │
   SSE Events           ┌────▼────┐
                        │  Model  │
                        │ Provider│
                        └─────────┘

Cost comparison (per 1M tokens):

Model	Input	Output	Cache Read	Best For
Haiku 4.5	$1.00	$5.00	$0.10	Simple queries, greetings
Sonnet 4.5	$3.00	$15.00	$0.30	Security audits, analysis
Opus 4.5	$5.00	$25.00	$0.50	Complex reasoning

With routing + caching, typical security audit costs ~$0.08-0.15 vs $0.40+ without.

Conclusion

The Strands TypeScript SDK provides a solid foundation for building production AI agents. Key takeaways:

Tools are everything - Design domain-specific tools, not generic utilities
Cache aggressively - Message caching saves 90% on repeated context
Route by complexity - Not every request needs your best model
Stream for UX - Users need to see progress during long operations
Track costs - Log token usage and compare models in production

The SDK handles the agentic loop so you can focus on domain logic. For ai-secure.dev, that meant security analysis - not prompt engineering infrastructure.

Questions or building your own agent? Connect on LinkedIn.

AWS Bedrock AgentCore - AI Agent Development from Local to Cloud

Fri, 09 Jan 2026 00:00:00 GMT

Introduction

Building production-ready AI agents requires more than just a prompt and an LLM. You need infrastructure for state management, tool execution, and secure deployment. I recently built https://ai-secure.dev, a SaaS for automated security compliance audits, using AWS Bedrock AgentCore. This post explores how AgentCore simplifies the transition from a local prototype to a scalable cloud agent.

What is AWS Bedrock AgentCore?

AgentCore is AWS's managed runtime for AI agents. Think of it as "Fargate for AI agents" - you bring your container, AWS handles scaling, networking, and infrastructure.

Key features:

Container-based: Package agent as Docker image, push to ECR, deploy via CDK
VPC networking: Agents run in private subnets with NAT for outbound (Anthropic API, target websites)
AgentCore Browser: Managed Chromium browser for web automation - no Playwright/Puppeteer infra needed
Memory: Built-in conversation memory across sessions
Streaming: SSE responses for real-time progress updates

https://ai-secure.dev

A security compliance scanner that uses an AI agent to audit websites.

How it works:

User submits URL + selects security compliance framework (ISO 27001, NIST, SOC2, COBIT)

Agent navigates site using AgentCore Browser, shows real-time progress

Scan completes with summary

User views detailed report with findings + recommendations

Architecture:

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│   Frontend  │────▶│  AgentCore       │────▶│  AgentCore  │
│  (Next.js)  │     │  Runtime         │     │   Browser   │
└─────────────┘     │  (your container)│     │  (Chromium) │
                    └──────────────────┘     └─────────────┘
                           │
                     ┌─────▼─────┐
                     │  Claude   │
                     │ (Anthropic)│
                     └───────────┘

Tech stack:

Component	Tech
Agent	Strands TypeScript SDK + Claude (Anthropic)
Browser	AgentCore Browser (cloud) / Playwright MCP (local)
Frontend	Next.js 16 + React 19 + Tailwind (Hosted on AWS ECS Fargate)
Auth	AWS Cognito
Database	DynamoDB
Infra	AWS CDK → AWS ECS Fargate + AWS Bedrock AgentCore Runtime
Payments	Stripe

Agent tools:

browser_navigate, browser_snapshot, browser_click, browser_type - web automation
http_security_check - headers, TLS, redirects
dns_lookup - SPF/DMARC/CAA records
totp - 2FA code generation for authenticated scans
issue_tracker - tracks problems during scan

Model routing for cost optimization:

// Simple tasks → Haiku (~10x cheaper)
// Complex tasks → Sonnet
const classification = classifyTask(prompt)
const model = classification.complexity === 'simple' 
  ? 'claude-haiku-4-5' 
  : 'claude-sonnet-4-5'

Learnings

Agent development:

Don't build an agent from scratch initially. Validate your use case works with existing agents (Cursor, Claude Code, Kiro). If they can't solve it, your custom agent probably won't either. Your goal: beat them on speed, accuracy, cost for your specific domain.
Develop locally first using Docker. Same container runs locally and in AgentCore.
Use Server-Sent Events (SSE) streaming for real-time progress - users need to see what the agent is doing.

Browser automation:

AgentCore Browser is game-changer. No more managing Playwright/Puppeteer infrastructure.
For local dev, use existing MCP Docker images (playwright-mcp). Your custom implementation won't be better.
Browser tools need good error handling - pages don't always load, elements move, auth flows vary.

Cost optimization:

Model routing: Haiku for greetings/simple queries, Sonnet for audits
Message caching: 90% cost reduction on repeated context
Disable extended thinking unless needed (~$0.15/call)

Production gotchas:

AgentCore logs are separate from your app logs. Use CloudWatch SDK directly for application logging.
Use Infrastructure as Code (IaC) to deploy your agent to AgentCore.

Conclusion

AWS Bedrock AgentCore provides a robust foundation for building complex, stateful AI agents. By offloading the heavy lifting of runtime management and browser infrastructure to AWS, I was able to focus on the core logic of https://ai-secure.dev - creating high-quality security audits - rather than debugging infrastructure. If you're building agents that need to browse the web or maintain long-term state, AgentCore is a powerful accelerator.

Please give me feedback on LinkedIn. Either if you find the https://ai-secure.dev useful and how I can make it better or if you need help with your own AI agent project. More and more builders are curious about how I build and deploy AI Agents in more specifics. So let me know if you would interested in more blog posts around this or perhaps a workshop/course around how I build my AI Agents?!

As well I think my next post will be about Agent Skills as I'm exploring those to start another AI Agent idea where I want the agent helping me with marketing for https://ai-secure.dev. So make sure to subscribe to my blog posts RSS feed.

A MCP Server with AWS Bedrock AgentCore Gateway

Sat, 20 Sep 2025 00:00:00 GMT

Hi there,

I recently developed an MCP Server with using our HalloCasa Zapier Integration.

Zapier MCP is nice but I wanted to try AWS Bedrock AgentCore Gateway to develop an MCP Server
Because it allows you to leverage an OpenAPI Spec as target which we already have
I was curious to see how it would perform
Additionally adjusting our openapi spec to serve better as an MCP Server improves the the spec

Conclusion

...

If you want more details, like how I build the MCP integration or how I deploy the NextJS app via AWS CDK, reach out to me on LinkedIn. I'm super eager to share my knowledge with you.

A property listings AI chat with MCP integrations running in AWS

Wed, 09 Jul 2025 00:00:00 GMT

Hi there,

My Co-Founder Michael Franz and I founded https://hallocasa.com a while back. It is a property listing and agency platform to list and search for properties https://hallocasa.com/properties or to find a property broker https://hallocasa.com/brokers.

As I'm a big fan of AWS, AI and MCP (Model Context Protocol https://modelcontextprotocol.io/), I wanted to build an AI chat with MCP integrations running in AWS to improve the property search experience. I also believe the future of property search will be AI-driven and you basically write or speak to an AI to find a property.

listings-mcp.com

I'm hosting the app on https://listings-mcp.com and you can try it out there. It is comparable with https://www.perplexity.ai/ when you use it for property search. But perplexity isn't working as good as I expected. It pretends to show properties but if you click on the source it only forwards you to the properties search. But what I really want is to see the property details and images like you get with https://listings-mcp.com.

MCP Integration

We do have a HalloCasa Zapier Integration. That is quite convenient because Zapier allows you to use your integration as MCP server https://zapier.com/mcp. So very conveniently with Zapier MCP you choose the Zapier integration like HalloCasa, choose which tools you want to use and configure the parameters (if you need at all). After that Zapier provides you with an MCP server url which you can use in your AI chat.

This approach is incredible easy and powerful as it allows me to enhance the MCP quickly.

For putting together the AI Chat capabilities and the MCP integration, I'm using the AI libraries from Vercel ai-sdk. The mcp-tools make it incredible easy to build an AI chat with MCP integrations. I wrote everything down in a few hours. Of course cursor AI made most of the work.

Here is the code for the nextJS page route /api/chat:

import { openai } from "@ai-sdk/openai"
import { createAmazonBedrock } from "@ai-sdk/amazon-bedrock"
import { fromNodeProviderChain } from "@aws-sdk/credential-providers"
import { streamText, experimental_createMCPClient } from "ai"
import { Experimental_StdioMCPTransport } from "ai/mcp-stdio"
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"
import { getEnabledMCPServers, validateMCPConfig, type MCPServerConfig } from "@/lib/mcp-config"
import { DynamoDBClient } from "@aws-sdk/client-dynamodb"
import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb"
import { randomUUID } from "crypto"

export const maxDuration = 30

// Model configuration - you can switch between providers here
const MODEL_PROVIDER = process.env.MODEL_PROVIDER || "bedrock" // "openai" or "bedrock"

// DynamoDB configuration
const dynamoDBClient = new DynamoDBClient({ 
  region: process.env.AWS_REGION || "us-east-1",
  ...(process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY ? {
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
      ...(process.env.AWS_SESSION_TOKEN && { sessionToken: process.env.AWS_SESSION_TOKEN })
    }
  } : {
    credentialProvider: fromNodeProviderChain()
  })
})
const docClient = DynamoDBDocumentClient.from(dynamoDBClient)
const CHAT_SESSIONS_TABLE = process.env.DYNAMODB_CHAT_SESSIONS_TABLE || "listings-mcp-chat-sessions"

// Helper functions for session management
interface ChatMessage {
  sessionId: string
  messageId: string
  role: 'user' | 'assistant'
  content: string
  timestamp: number
  userId?: string
  includePropertyPictures?: boolean
  toolCalls?: any[]
  toolResults?: any[]
  metadata?: any
  ttl?: number // TTL for automatic cleanup (30 days from now)
}

function generateSessionId(): string {
  return randomUUID()
}

function generateMessageId(): string {
  return randomUUID()
}

function getUserId(req: Request): string {
  // Try to extract user ID from headers, auth token, or IP address
  const userAgent = req.headers.get('user-agent') || 'unknown'
  const forwarded = req.headers.get('x-forwarded-for') || req.headers.get('x-real-ip') || 'unknown'
  
  // For now, create a simple hash-based user ID
  // In production, this should come from authentication
  const userIdentifier = `${userAgent}-${forwarded}`.replace(/[^a-zA-Z0-9-]/g, '_')
  return userIdentifier.substring(0, 50) // Limit length
}

async function saveMessageToDynamoDB(message: ChatMessage): Promise<void> {
  try {
    const ttl = Math.floor(Date.now() / 1000) + (30 * 24 * 60 * 60) // 30 days from now
    
    await docClient.send(new PutCommand({
      TableName: CHAT_SESSIONS_TABLE,
      Item: {
        ...message,
        ttl
      }
    }))
    
    console.log(`[DynamoDB] Saved message ${message.messageId} to session ${message.sessionId}`)
  } catch (error) {
    console.error('[DynamoDB] Error saving message:', error)
    // Don't throw - we don't want to break the chat if DB fails
  }
}

// Create Bedrock provider with region configuration
// In ECS, credentials are automatically provided via the task role
// Only set explicit credentials if they exist (for local development)
const bedrockConfig: any = {
  region: process.env.AWS_REGION || "us-east-1", // Default to us-east-1 if not set
}

// Check if explicit credentials are provided (for local development)
if (process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY) {
  bedrockConfig.accessKeyId = process.env.AWS_ACCESS_KEY_ID
  bedrockConfig.secretAccessKey = process.env.AWS_SECRET_ACCESS_KEY
  if (process.env.AWS_SESSION_TOKEN) {
    bedrockConfig.sessionToken = process.env.AWS_SESSION_TOKEN
  }
} else {
  // Use AWS credential provider chain for automatic credential resolution
  // This will work with ECS task roles, EC2 instance profiles, etc.
  bedrockConfig.credentialProvider = fromNodeProviderChain()
}

const bedrock = createAmazonBedrock(bedrockConfig)

function getModel() {
  switch (MODEL_PROVIDER) {
    case "openai":
      return openai("gpt-4o")
    case "bedrock":
    default:
      // Use inference profile ID for cross-region access to Claude Sonnet 4
      return bedrock("us.anthropic.claude-sonnet-4-20250514-v1:0")
  }
}

async function connectToMCPServers(enabledServers: MCPServerConfig[]): Promise<{ clients: any[], allTools: any }> {
  const clients: any[] = []
  let allTools = {}
  
  console.log(`[MCP] Attempting to connect to ${enabledServers.length} MCP servers...`)
  
  for (const serverConfig of enabledServers) {
    console.log(`[MCP] Processing server: ${serverConfig.name} (${serverConfig.type})`)
    
    // Validate configuration before attempting connection
    if (!validateMCPConfig(serverConfig)) {
      console.warn(`[MCP] Invalid MCP server configuration for ${serverConfig.name}`)
      continue
    }
    try {
      console.log(`[MCP] Creating client for ${serverConfig.name}...`)
      let client

      switch (serverConfig.type) {
        case 'sse':
          client = await experimental_createMCPClient({
            transport: {
              type: 'sse',
              url: serverConfig.url!
            }
          })
          break
        
        case 'http':
          const transport = new StreamableHTTPClientTransport(new URL(serverConfig.url!))
          client = await experimental_createMCPClient({
            transport
          })
          break
          
        case 'stdio':
          console.log(`[MCP] Creating stdio transport for ${serverConfig.name} with command: node ${serverConfig.command}`)
          const stdioTransport = new Experimental_StdioMCPTransport({
            command: 'node',
            args: [serverConfig.command!]
          })
          console.log(`[MCP] Creating MCP client for ${serverConfig.name}...`)
          client = await experimental_createMCPClient({
            transport: stdioTransport
          })
          console.log(`[MCP] Client created for ${serverConfig.name}, waiting for initialization...`)
          // Give stdio process more time to fully initialize
          await new Promise(resolve => setTimeout(resolve, 2000))
          console.log(`[MCP] Initialization wait complete for ${serverConfig.name}`)
          break
      }

      if (client) {
        clients.push({ client, config: serverConfig })
        
        // Get tools from this MCP server with retry logic
        let serverTools: any = {}
        let attempts = 0
        const maxAttempts = 3
        
        while (attempts < maxAttempts) {
          try {
            console.log(`[MCP] Attempt ${attempts + 1} to get tools from ${serverConfig.name}...`)
            serverTools = await client.tools()
            console.log(`[MCP] Successfully retrieved ${Object.keys(serverTools).length} tools from ${serverConfig.name}`)
            break // Success, exit retry loop
          } catch (toolError) {
            attempts++
            console.warn(`[MCP] Attempt ${attempts} failed to get tools from ${serverConfig.name}:`, toolError)
            if (attempts < maxAttempts) {
              // Wait before retrying (exponential backoff)
              await new Promise(resolve => setTimeout(resolve, 500 * attempts))
            } else {
              console.error(`[MCP] Failed to get tools from ${serverConfig.name} after ${maxAttempts} attempts`)
              throw toolError
            }
          }
        }
        
        // Prefix tool names with server name to avoid conflicts
        const prefixedTools = Object.keys(serverTools).reduce((acc, toolName) => {
          const prefixedName = `${serverConfig.name}_${toolName}`
          const originalTool = serverTools[toolName]
          
          // Use the original tool directly from MCP client
          // The AI SDK should handle MCP tools correctly without modification
          acc[prefixedName] = originalTool
          console.log(`[MCP] Added tool ${prefixedName} with original MCP interface`)
          return acc
        }, {} as any)
        
        allTools = { ...allTools, ...prefixedTools }
        
        console.log(`[MCP] Successfully connected to MCP server: ${serverConfig.name} with ${Object.keys(serverTools).length} tools`)
      }
    } catch (error) {
      console.warn(`[MCP] Failed to connect to MCP server ${serverConfig.name}:`, error)
      // Continue with other servers even if one fails
    }
  }

  console.log(`[MCP] Connection process complete. Connected to ${clients.length} servers with ${Object.keys(allTools).length} total tools`)
  return { clients, allTools }
}

export async function POST(req: Request) {
  const { messages, includePropertyPictures, sessionId: existingSessionId } = await req.json()
  console.log(`[API] Received chat request with ${messages.length} messages`)
  console.log(`[API] Include property pictures: ${includePropertyPictures}`)
  console.log(`[API] Existing session ID: ${existingSessionId || 'none'}`)

  // Generate or use existing session ID
  const sessionId = existingSessionId || generateSessionId()
  const userId = getUserId(req)
  const timestamp = Date.now()

  // Track tool usage for this session
  const toolUsageTracker = {
    totalSteps: 0,
    toolCalls: [] as any[],
    toolResults: [] as any[]
  }

  console.log(`[API] Session ID: ${sessionId}, User ID: ${userId}`)

  // Save user messages to DynamoDB (only new messages)
  if (messages.length > 0) {
    const lastMessage = messages[messages.length - 1]
    if (lastMessage.role === 'user') {
      const userMessage: ChatMessage = {
        sessionId,
        messageId: generateMessageId(),
        role: 'user',
        content: lastMessage.content,
        timestamp,
        userId,
        includePropertyPictures,
        metadata: {
          userAgent: req.headers.get('user-agent'),
          ip: req.headers.get('x-forwarded-for') || req.headers.get('x-real-ip')
        }
      }
      
      // Save user message (don't await to avoid blocking)
      saveMessageToDynamoDB(userMessage).catch(console.error)
    }
  }

  try {
    // Get enabled MCP servers
    const enabledServers = getEnabledMCPServers()
    console.log(`[API] Found ${enabledServers.length} enabled MCP servers`)
    
    // Connect to MCP servers and get tools
    const startTime = Date.now()
    const { clients, allTools } = await connectToMCPServers(enabledServers)
    const connectionTime = Date.now() - startTime
        console.log(`[API] MCP connection process took ${connectionTime}ms`)
    console.log(`[API] Available tools:`, Object.keys(allTools))

    console.log('[API] Starting streamText...')
    console.log('[API] Tool count:', Object.keys(allTools).length)
    console.log('[API] Tool names:', Object.keys(allTools))
    
    // Build system prompt with conditional instructions
    const pictureInstruction = includePropertyPictures 
      ? "- When returning property data, make sure to include the first picture\n"
      : ""
    
    const result = streamText({
      model: getModel(),
      system: `You are a knowledgeable international real estate assistant helping clients find properties worldwide. You have access to various property platforms and tools through MCP (Model Context Protocol) integrations.

You should:
- Help clients explore property options in different countries
- Use MCP tools to fetch real-time property data when available
- Provide insights about local real estate markets, prices, and trends
- Discuss legal requirements for foreign property buyers
- Suggest popular areas and neighborhoods
- Explain visa/residency requirements related to property ownership
- Offer guidance on property types, investment potential, and lifestyle factors
- Be friendly, professional, and thorough in your responses
- Ask clarifying questions to better understand their needs
- When returning property data, include the source and the property url of the data in the response.
${pictureInstruction}
When using MCP tools, mention which platform the data is coming from to build trust and transparency.

Available MCP integrations: ${enabledServers.map((s: MCPServerConfig) => `${s.name} (${s.description})`).join(', ')}

IMPORTANT: When users ask about properties or real estate data, you MUST use the available MCP tools to fetch current data. Always try to use tools like demo_search_properties, demo_get_market_data, or demo_get_property_details when relevant. The tool results will contain the property data you need to include in your response.

Always be helpful and informative while encouraging them to consult with local real estate professionals for specific transactions.`,
      messages,
      tools: allTools,
      maxRetries: 3,
      maxSteps: 3, // Allow multiple steps for tool execution and response
      onStepFinish: async ({ text, toolCalls, toolResults, finishReason, usage }) => {
        toolUsageTracker.totalSteps++
        
        console.log(`[AI] Step ${toolUsageTracker.totalSteps} finished:`)
        console.log(`[AI]   - Text generated: ${text ? text.substring(0, 100) + '...' : 'none'}`)
        console.log(`[AI]   - Tool calls: ${toolCalls?.length || 0}`)
        
        if (toolCalls && toolCalls.length > 0) {
          toolCalls.forEach((call, idx) => {
            console.log(`[AI]   - Tool ${idx + 1}: ${call.toolName} with args:`, call.args)
            
            // Track tool calls with step information
            toolUsageTracker.toolCalls.push({
              step: toolUsageTracker.totalSteps,
              toolCallId: call.toolCallId,
              toolName: call.toolName,
              args: call.args,
              timestamp: Date.now()
            })
          })
        }
        
        console.log(`[AI]   - Tool results: ${toolResults?.length || 0}`)
        if (toolResults && toolResults.length > 0) {
          toolResults.forEach((result, idx) => {
            console.log(`[AI]   - Result ${idx + 1}:`, JSON.stringify(result, null, 2))
            
            // Track tool results with step information
            toolUsageTracker.toolResults.push({
              step: toolUsageTracker.totalSteps,
              toolCallId: result.toolCallId,
              toolName: result.toolName,
              result: result.result,
              timestamp: Date.now()
            })
          })
        }
        console.log(`[AI]   - Finish reason:`, finishReason)
      },
      onFinish: async (result) => {
        console.log(`[AI] Chat finished. Tool calls made:`, result.toolCalls?.length || 0)
        if (result.toolCalls && result.toolCalls.length > 0) {
          result.toolCalls.forEach((toolCall, index) => {
            console.log(`[AI] Tool call ${index + 1}:`, { 
              name: toolCall.toolName, 
              args: toolCall.args
            })
           })
        }
        console.log(`[AI] Final response length:`, result.text.length)
        console.log(`[AI] Response sample:`, result.text.substring(0, 200) + '...')
        
        // Save assistant response to DynamoDB
        const assistantMessage: ChatMessage = {
          sessionId,
          messageId: generateMessageId(),
          role: 'assistant',
          content: result.text,
          timestamp: Date.now(),
          userId,
          includePropertyPictures,
          toolCalls: toolUsageTracker.toolCalls,
          toolResults: toolUsageTracker.toolResults,
          metadata: {
            usage: result.usage,
            finishReason: result.finishReason,
            warnings: result.warnings,
            modelProvider: MODEL_PROVIDER,
            totalSteps: toolUsageTracker.totalSteps,
            toolUsageSummary: {
              totalToolCalls: toolUsageTracker.toolCalls.length,
              uniqueTools: [...new Set(toolUsageTracker.toolCalls.map(tc => tc.toolName))],
              callsPerTool: toolUsageTracker.toolCalls.reduce((acc, tc) => {
                acc[tc.toolName] = (acc[tc.toolName] || 0) + 1
                return acc
              }, {} as Record<string, number>)
            }
          }
        }
        
        // Save assistant message (don't await to avoid blocking cleanup)
        saveMessageToDynamoDB(assistantMessage).catch(console.error)
        
        // Clean up MCP clients when streaming is finished
        for (const { client } of clients) {
          try {
            await client.close()
          } catch (error) {
            console.warn('Error closing MCP client:', error)
          }
        }
      },
      onError: async (errorObj) => {
        console.error('[AI] Streaming error occurred:', {
          error: errorObj.error,
          errorString: String(errorObj.error)
        })
        // Clean up MCP clients on error
        for (const { client } of clients) {
          try {
            await client.close()
          } catch (closeError) {
            console.warn('Error closing MCP client on error:', closeError)
          }
        }
      }
    })

    return result.toDataStreamResponse()
  } catch (error) {
    console.error('Failed to initialize chat with MCP:', error)
    
    // Build system prompt with conditional instructions for fallback
    const pictureInstruction = includePropertyPictures 
      ? "- When returning property data, make sure to include the first picture\n      "
      : ""
    
    // Fallback to basic chat without MCP if initialization fails
    const result = streamText({
      model: getModel(),
      system: `You are a knowledgeable international real estate assistant helping clients find properties worldwide.

You should:
      
      - Help clients explore property options in different countries
      - Provide insights about local real estate markets, prices, and trends
      - Discuss legal requirements for foreign property buyers
      - Suggest popular areas and neighborhoods
      - Explain visa/residency requirements related to property ownership
      - Offer guidance on property types, investment potential, and lifestyle factors
      - Be friendly, professional, and thorough in your responses
      - Ask clarifying questions to better understand their needs
      ${pictureInstruction}
      Note: Advanced property platform integrations are temporarily unavailable.
      
      Always be helpful and informative while encouraging them to consult with local real estate professionals for specific transactions.`,
      messages,
      onFinish: async (result) => {
        console.log(`[AI] Fallback chat finished. Response length:`, result.text.length)
        
        // Save assistant response to DynamoDB (fallback mode)
        const assistantMessage: ChatMessage = {
          sessionId,
          messageId: generateMessageId(),
          role: 'assistant',
          content: result.text,
          timestamp: Date.now(),
          userId,
          includePropertyPictures,
          metadata: {
            usage: result.usage,
            finishReason: result.finishReason,
            warnings: result.warnings,
            modelProvider: MODEL_PROVIDER,
            mcpFallback: true
          }
        }
        
        // Save assistant message (don't await to avoid blocking)
        saveMessageToDynamoDB(assistantMessage).catch(console.error)
      }
    })

    return result.toDataStreamResponse()
  }
}

AWS Setup

I'm using AWS CDK to deploy AWS ECS Fargate to run a NextJS app which has access to the MCP server and Anthropic Claude 4 Sonnet via AWS Bedrock. The nextjs app is configured as NextJS standalone app.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ecsPatterns from 'aws-cdk-lib/aws-ecs-patterns';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as route53 from 'aws-cdk-lib/aws-route53';
import * as acm from 'aws-cdk-lib/aws-certificatemanager';

export interface ListingsMcpServiceStackProps extends cdk.StackProps {
  cluster: ecs.Cluster;
}

export class ListingsMcpServiceStack extends cdk.Stack {
  public readonly fargateService: ecsPatterns.ApplicationLoadBalancedFargateService;

  constructor(scope: Construct, id: string, props: ListingsMcpServiceStackProps) {
    super(scope, id, props);

    const domainName = 'listings-mcp.com';

    // Look up the existing hosted zone for the domain
    const hostedZone = route53.HostedZone.fromLookup(this, 'HostedZone', {
      domainName,
    });

    // Create an ACM certificate for the domain
    const certificate = new acm.Certificate(this, 'Certificate', {
      domainName,
      validation: acm.CertificateValidation.fromDns(hostedZone),
    });

    // Create a Fargate service with an Application Load Balancer
    this.fargateService = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'ListingsMcpService', {
      cluster: props.cluster,
      memoryLimitMiB: 512,
      cpu: 256,
      desiredCount: 1,
      enableExecuteCommand: true,
      certificate: certificate,
      domainName,
      domainZone: hostedZone,
      redirectHTTP: true,
      healthCheck: {
        command: ['CMD-SHELL', 'curl -f http://localhost:3000/ || exit 1'],
      },
      
      runtimePlatform: {
        cpuArchitecture: ecs.CpuArchitecture.ARM64,
        operatingSystemFamily: ecs.OperatingSystemFamily.LINUX,
      },
      taskImageOptions: {
        image: ecs.ContainerImage.fromAsset('..'),
        containerPort: 3000,
        environment: {
          NODE_ENV: 'production',
          NEXT_TELEMETRY_DISABLED: '1',
          HOSTNAME: '0.0.0.0',
          TRUST_PROXY: '1',
          AWS_REGION: this.region,
          NEXT_PUBLIC_BASE_URL: '/'
        },
      },
    });

    // Add SES permissions to the task role for sending verification emails
    (this.fargateService.taskDefinition.taskRole as iam.Role).addToPolicy(new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      actions: [
        'ses:SendEmail',
        'ses:SendRawEmail',
      ],
      resources: [
        `arn:aws:ses:${this.region}:${this.account}:identity/*`,
      ],
    }));

    // Add Cognito permissions to the task role for user management operations
    (this.fargateService.taskDefinition.taskRole as iam.Role).addToPolicy(new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      actions: [
        'cognito-idp:ListUsers',
        'cognito-idp:AdminCreateUser',
        'cognito-idp:AdminUpdateUserAttributes',
        'cognito-idp:AdminGetUser',
        'cognito-idp:AdminSetUserPassword',
        'cognito-idp:AdminInitiateAuth',
        'cognito-idp:InitiateAuth',
        'cognito-idp:RespondToAuthChallenge',
      ],
      resources: [
        `arn:aws:cognito-idp:${this.region}:${this.account}:userpool/${props.userPoolId}`,
      ],
    }));

    // Add STS permissions for getting AWS account information
    (this.fargateService.taskDefinition.taskRole as iam.Role).addToPolicy(new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      actions: [
        'sts:GetCallerIdentity',
      ],
      resources: ['*'],
    }));

    // Add CloudWatch Logs permissions for better logging and debugging
    (this.fargateService.taskDefinition.taskRole as iam.Role).addToPolicy(new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      actions: [
        'logs:CreateLogGroup',
        'logs:CreateLogStream',
        'logs:PutLogEvents',
        'logs:DescribeLogStreams',
      ],
      resources: [
        `arn:aws:logs:${this.region}:${this.account}:log-group:/aws/ecs/*`,
      ],
    }));

    // Add Bedrock permissions for AI model access
    (this.fargateService.taskDefinition.taskRole as iam.Role).addToPolicy(new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      actions: [
        'bedrock:InvokeModel',
        'bedrock:InvokeModelWithResponseStream',
      ],
      resources: [
        `arn:aws:bedrock:*:${this.account}:inference-profile/*`,
        `arn:aws:bedrock:*::foundation-model/*`,
      ],
    }));

    // Add base URL environment variable after service creation
    this.fargateService.taskDefinition.defaultContainer?.addEnvironment(
      'NEXT_PUBLIC_BASE_URL',
      `https://${domainName}`
    );

    // Output the load balancer URL
    new cdk.CfnOutput(this, 'LoadBalancerDNS', {
      value: this.fargateService.loadBalancer.loadBalancerDnsName,
      description: 'The DNS name of the load balancer',
    });

    // Output the custom domain URL
    new cdk.CfnOutput(this, 'ApplicationURL', {
      value: `https://${domainName}`,
      description: 'The URL of the application',
    });

    // Output the certificate ARN
    new cdk.CfnOutput(this, 'CertificateArn', {
      value: certificate.certificateArn,
      description: 'The ARN of the SSL certificate',
    });
  }
}

Conclusion

So try out https://listings-mcp.com and let me know what you think. I'm curious if we can improve the property search experience by particular using MCP integrations from providers like HalloCasa. I'm not that crazy of trying to compete with perplexity but maybe there is a niche for this. Maybe we can become a market place specifically for property MCP integrations.

If you need help with MCP integrations, let me know. I'm happy to help you.

If you want more details, like how I build the MCP integration or how I deploy the NextJS app via AWS CDK, reach out to me on LinkedIn. I'm super eager to share my knowledge with you.

SST, AWS CDK, AWS CloudFormation migration to Terraform

Tue, 27 Aug 2024 00:00:00 GMT

Recently, a client approached me with an intriguing request: to migrate their SST (v.2 https://v2.sst.dev/) project to Terraform. This task presents several interesting challenges, particularly in mapping SST and CDK constructs to equivalent Terraform resources. Additionally, certain components like Lambda functions and static S3 React buckets require extra attention, as they involve uploading additional files such as Lambda function code or React builds.

Although I performed this migration specifically for SST, the process I'll outline is equally applicable to AWS CDK or AWS CloudFormation projects. If you're facing a similar challenge, this blog post aims to provide you with a solid starting point and valuable insights into the migration process.

First, let's discuss some possible motivations and non-motivations for migrating from SST, AWS CDK, or AWS CloudFormation to Terraform.

After that, I will describe the migration process in detail.

Motivations

My client's motivation was rather passive. As a newcomer to AWS, he used SST (v.2) to quickly deploy his application, which worked well for him. However, his company eventually decided to standardize on Terraform instead of SST. This highlights alignment as a motivation: if your company is using Terraform, you probably should too.

Another commonly cited reason is Terraform's superior state management. While Terraform deployments are faster than SST or AWS CDK, the difference doesn't feel significant to me.

There are undoubtedly more motivations; feel free to mention important ones I may have missed.

Non-Motivations

The following are reasons I wouldn't consider as motivations for migrating to Terraform:

The biggest one is the degree of abstraction, like as for the CDK Constructs. You would lose that when migrating to Terraform. Defining many basic resources in Terraform isn't particularly enjoyable. To counter this somewhat, you can use an AI tool in your IDE to speed up the definition process. As well, Terraform modules give you a way to abstract definitions to modules, but it doesn't feel the same as the CDK Constructs.

As SST and AWS CDK are already TypeScript-based, they make it really easy to define and handle the lifecycle of Lambda functions. In Terraform, you need to define helper functions to manage the Lambda function lifecycle, such as bundling the code, creating the deployment package, and so on. This was actually quite painful for my project, and I ended up with this rather inelegant script:

rm -rf lambda_function_payload.zip
cd ../functions
rm -rf dist
npx tsc
npm install
cp -r node_modules/ dist/node_modules
cd dist
zip -rFS lambda_function_payload.zip *
cp -r lambda_function_payload.zip ../../../terraform
cd ../../../terraform

This feels ugly and was no fun.

Kind of similar is the deployment of a React app into a S3 bucket. In SST and AWS CDK you can use the Bucket construct and let the framework handle the deployment. In Terraform, you need to manually deploy the React app to the S3 bucket and then invalidate the CloudFront cache.

Migration

In the next section, I will describe how the migration from SST, AWS CDK, or AWS CloudFormation to Terraform works.

Step 1: Deploy

Before you start the migration, make sure your deployment works as expected. If you have an SST project, follow the instructions for deploying it like npx sst deploy. After deploying, check the functionality of the CloudFormation stacks. That is super important as that will be your comparison to the Terraform deployment.

Step 2: Generate Terraform from the CloudFormation template

Now that your deployment from step 1 created at least one CloudFormation stack, you can start the migration.

Via the AWS Console, grab each generated stack and let a chat AI like Claude from anthropic.com or ChatGPT.com generate Terraform from the CloudFormation template. The prompt could look like:

{
  "Resources": {
    "CustomResourceHandlerServiceRole41AEC181": {
      "Type": "AWS::IAM::Role",
      ...
}

Change the AWS CloudFormation to Terraform. Give back the full Terraform code!

Through the output from all those answers into the main.tf file.

Step 3: Cleanup the main.tf

Great now we have all the AWS resources in the main.tf file. But unlucky we have resources in the main.ts file which we don't want or which are not useful like those AWS CDK helper resources CustomResourceHandlerServiceRole... and CustomResourceHandler....

As well, there are might be other resources which you might consider removing. For example, if you have several stacks and used variable referencing between the CloudFormation stacks, the AI translation to Terraform usually uses aws_ssm_parameter Terraform resources to replace them. But since all your resources are in one main.tf file, you don't need aws_ssm_parameter and simply reference resources directly.

Step 4: Make the Lambda's working

Yeah, now comes the tricky part with making the Lambda's working. Look this totally depends on your project structure like where is the Lambda function code located and how is it structured. I think good advice is to keep it similar as possible to your SST or AWS CDK project. As well, use the source_code_hash to make sure the Lambda function code is hashed and the Lambda function is recreated if the code changes. Like:

resource "aws_lambda_function" "my_lambda_function" {
  ...
  source_code_hash = filebase64sha256("my_lambda_function.zip")
}

What is left is the script to bundle the Lambda function code to the my_lambda_function.zip file. For you, it could look something like this:

rm -rf lambda_function_payload.zip
cd ../functions
rm -rf dist # the tsconfig.json has an "outDir": "./dist" configured where all the compiled js files will be stored
npx tsc # compile the TypeScript code
npm install
cp -r node_modules/ dist/node_modules
cd dist
zip -rFS lambda_function_payload.zip *
cp -r lambda_function_payload.zip ../../terraform
cd ../../terraform

Deploy the Lambda function and check with the AWS Console if it is working as expected.

terraform apply

Step 5: Make the S3 React bucket working

Sure this is described for a React App but any other SPA or static site will work the same. First, you need to build the React app like:

cd ../frontend
npm install
npm run build

Copy the build to the s3 bucket and invalidate the cloudfront cache:

BUCKET_NAME=xyz-react-site-bucket
DISTRIBUTION_ID=EZ8RBY8ZM1234
aws s3 sync dist/ s3://$BUCKET_NAME --delete

aws cloudfront create-invalidation --distribution-id $DISTRIBUTION_ID --paths "/*"

Phew, that's it.

Step 6: Validating

Validate the Terraform deployment with your reference deployment from step 1. Usually like when you have an Api Gateway our Lambdas, make sure they are working as expected when comparing the two deployments. The same goes for the S3 React bucket App.

Considerations

Sure, storing all those AWS resources into one main.tf file isn't Terraform best practice, but it is totally a starting point. If you made sure that your Terraform deployment is working, feel free to split the main.tf file into multiple files, as you are used to.

Conclusion

Migrating SST to Terraform was interesting. With the power of AI, it was a quick process. Combined with my years of experience, I was able to quickly migrate. If you have a question or need otherwise help, please reach out to me.

Bonus - AB Picturer

Did you notice the cool blog title picture? I love writing blog posts and choosing nice pictures for them. It is actually one of two randomly selected pictures. But often I want to choose THE BEST picture. So to find the best picture I'm using AB Testing. If you are curious about it, have a look at the https://ab-picturer.com tool and provide me feedback or even better become an engaged tester :).

Quiz

Deepen your knowledge about this blog post with a click on the AI generated quiz https://quizrun.ai/?url=https://martinmueller.dev/aws-sst-cdk-to-tf

HalloCasa Iframe test

Sun, 30 Jun 2024 00:00:00 GMT

Recertification - AWS Solution Architect Professional in 2024

Wed, 15 May 2024 00:00:00 GMT

Hi Friends.

My AWS Solution Architect Professional expires in mid-2022. Since then, I've occasionally thought about renewing, but I didn't want to invest the time to prepare. But now a customer was paying me to take the exam, so why not. Here's some advice that helped me minimize the effort and still pass the exam.

Learning for the exam

First of all, cherish learning! Learning is fun and studying for the exam is fun!

My favorite way to learn efficiently is to do mock exams like you get on Udemy or other learning platforms. For one, they prepare you for the test exam. And for others, it is a great way to have an amazing learning when you answer a question wrong. Usually the learning platform explains you exactly why your answer was wrong.

Learn in chunks

The main goal when using these test exams should be to maximize your learning with chunking your learning as this works better for humans. I don't think you should answer all 75 questions in one go. Rather, you should answer about 5, 10, or 15 so that when you review, you can see what you got wrong and write down what you learned. The goal is to get one or two questions wrong, which will give you a great learning opportunity. If you get all the questions right, you can increase the number of questions you answer in the next session or move on to the next quiz. This way you can have amazing learning every day.

Loud Sticks

Read the questions aloud and try to say important keywords like 'media company', 'cost effective' or 'availability' extra loud. I find that I pay more attention to the questions and answers when I do this. This is not only useful when preparing for the exam. It is also useful during the exam. But be cautious. The online exam will require silence from you. So what I did was whispering which was allowed.

So you've already maximized your learning opportunities through my tips, so let's move on to the next phase, registration and the actual exam.

Registration

First of all, do not be afraid of failing! Unfortunately failing is still not too accepted in our society, but failing is great as long as you learn from it and try to do better next time. Just look at my two cute little kids. They failed all the time, like when they were trying to walk or talk, but they never gave up and now they walk and talk like professionals.

So I suggest you to register for the test when you reach like 70% correct answers of a new test exams test like you get them on Udemy or other learning platforms. Don't wait until you get about 90% correct. The sooner you try, the better. Because even if you fail, you can analyze the results and see where you need to improve and just keep learning.

Sure, paying those 300 or what dollar for the exam isn't easy affordable for everyone. Yes, you have a fair point there! If that is difficult for you to do, try to get one of those vouchers to either reduce or completely eliminate the cost. For example, I know that the AWS Community Builder program gives you an exam voucher every year.

At home or not

When you register for the exam, you can choose to take it at home or at a testing facility. Both are great choices. I will choose home because I know I am most productive at home and I can have a quiet room for those 4 hours. Also, I know when my brain works best and that is after my morning workout and hot/cold shower. I'm going to use that little brain boost for the exam.

Celebration

Celebrate your exam journey! Celebration is fun and we usually do it to less. Celebrate your everyday learning. Celebrate the registration and celebrate the passing or failing of the exam.

Conclusion

Preparing and doing the exam is fun but! Enjoy the learnings during preparing for it. Tell me your tips for passing those exams :)!

Bonus - AB Picturer

Quiz

Deepen your knowledge about this blog post with a click on the AI generated quiz https://quizrun.ai/?url=https://martinmueller.dev/aws-cert-recert

AWS Bedrock Update from Claude v2.1 to Claude v3

Sun, 21 Apr 2024 00:00:00 GMT

Recently, AWS released Claude v3. It comes with the Haiku and Sonnet flavors. Both are big improvements over the previous version. We recently updated our arcBot to use Claude v3 Sonnet and are very impressed with the results. Not only are the responses more intelligent, but the speed of the responses is much faster. In the next section I will describe how an update can go as smoothly as it did for us.

Upgrade

As mentioned above, we upgraded from Claude v2.1 to Claude v3 Sonnet for our gen AI database tool arcBot (Try it out!).

When you do this there is a high risk that your answers will not be as good as before. But we were prepared and the answers are even better now. But how did we do it?

Simply with Unit Testing. I wrote a bunch of unit tests during the development with Claude v2.1 and they came in handy for the update to v3. It was basically just a matter of making those unit tests pass.

Conclusion

Using AWS gen AI offerings like Claude is super fun. It can be challenging to make sure your answers are still as good as before. But with good test coverage, you can make sure that your answers are still as good as before, or even better. With our unit tests in place, we are confident that we can easily upgrade to a future version again, or if we wanted, change the LLM entirely. If you have any questions or thoughts about this, feel free to contact us :)!

Bonus - AB Picturer

Did you notice the cool blog title picture? It is actually one of two randomly selected pictures. I love writing blog posts and choosing nice pictures for them. But often I want to choose THE BEST picture. So to find the best picture I'm using AB Testing. If you are curious about it, have a look at my AB Picturer Tool and provide me feedback or even better become an engaged tester :).

AI Software Development with Ninox and arcBot

Wed, 14 Feb 2024 00:00:00 GMT

AI is boosting the corporate landscape. Many simple and complex processes in companies can be greatly simplified or even completely automated using AI. Jakob, my co-founder, and I see it as an exciting challenge to simplify access to AI for small and medium-sized companies. We believe we have found an exciting approach with the low-code platform Ninox and our AI chatbot arcBot. In this blog post, I would like to introduce you to this approach.

Ninox.com

Ninox.com is a cloud-based low-code platform for building software solutions for small and medium-sized companies. Ninox was founded in 2013 in Germany by Frank Böhmer. Here are some ways Ninox can support businesses:

Business process automation: With Ninox, companies can automate their daily tasks and processes to increase efficiency.

Customizability: Because Ninox is a low-code platform, companies can adapt their applications to their specific needs without complex programming knowledge.

Integration: Ninox can be integrated with a variety of other tools and platforms, enabling seamless data exchange between different systems.

Cost efficiency: Compared to developing custom software from scratch, using a low-code platform like Ninox can result in significant cost savings.

Fast implementation: With Ninox, companies can build and implement their applications quickly, resulting in a faster time to market.

A few months ago, I created an MVP with Ninox. You can read about my experiences here. In the next chapter, I'll explain how we use arcBot to simplify the build for your software solution in ninox.

arcBot

The arcBot is a ChatGPT-like AI for quickly creating and modifying Ninox tables. We use the AWS Bedrock API to create the Ninox table responses. An example prompt to create Ninox tables could look like this:

Create customer and product tables in a many to many relationship.

Cool, right? arcBot creates many tables for customers and products. Remarkably, arcBot can even set the ninox-typical inverse fields. In the next prompt we want to add the last name to a customer.

Add last name to customer table

For the response, arcBot uses the Claude v2.1 model, which is made available via the AWS Bedrock API.

How do we ensure that the quality of the answers is as high as possible? We have built in a feedback loop. The user can give feedback after the answer. This feedback is then used to improve arcBot.

I am also in close contact with other Gen AI experts who help me to improve the quality of arcBot answers. I would like to take this opportunity to thank these experts. They are mainly AWS community members and AWS employees who have already given me many valuable tips. Thank you very much 🙏

Ninox Connector

It's cool that we can create Ninox tables with arcBot, but how can we process the data in Ninox? This is where the Ninox Connector comes into play. The Ninox Connector can read and update Ninox tables. The following image shows the Ninox Connector.

The Ninox Connector needs some information like the Ninox Url, Team Id, Database Id and the Ninox API Key. Currently the Ninox Connector is only available via Early Access. But you can easily get it on https://app.arcbot.de/.

Discord Community

Become part of our Discord Community. We already have some members who give us valuable feedback and feature requests for the arcBot.

Summary

In this blog post I have shown you how we simplify the creation and updating of Ninox tables by combining Ninox and arcBot. We look forward to your feedback. Thank you for reading.

I am passionate about contributing to Open Source projects. You can find many of my projects on GitHub that you can already benefit from.

If you found this post valuable and would like to show your support, consider supporting me back. Your support will enable me to write more posts like this and work on projects that provide value to you. You can support me by:

And don't forget to visit my site

Make your picture rock with the AB Picturer

Sat, 06 Jan 2024 00:00:00 GMT

Images are crucial to effective advertising! AI has made it incredibly easy to create images. But how do you know if an image will move your audience or not? You can try AB Picture Testing. With AB Picture Testing, two different image variants are used and pitted against each other. The one that performs better will be selected to increase traffic. To illustrate this, take a look at the image below, where the audience is separately looking at two different image variants.

My AB Picture Use Case

I like to write technical blog posts on my [blog] (https://martinmueller.dev). For these posts, I like to use an image that can be used as the thumbnail and main image for the post.

Creating images has become incredibly easy with tools like ChatGPT. By simply prompting it with a request like "Make a cute cloud that sings "I've got the power" while maintaining a ratio of 1000:420", you can create beautiful images that are perfect for sharing your posts on social media.

I am building a tool called AB Picturer. It allows me to measure the performance of the image variations. In the next section I explain how it works.

How does AB Picturer work?

The AB Picturer uses image urls like https://api.ab-picturer.com?projectId=ab&state=visitor to collect information like when the image was previewed and when the user decided to click on the post. This process is called click-through. AB Picturer can then calculate the click-through rate (CTR) for each image. The image with the higher CTR wins and can be used as the main image for the post.

The basic concept is to have two image urls that are part of the AB Picturer flow. The first is for the image preview and the second is for the image visitor. I'll describe these in more detail now.

AB Picturer Flow

The preview url is the beginning of the AB Picturer flow. When the user views the preview url via your thumbnail or social media post, the flow starts. The user session is tracked and if the user uses the image visitor url, this counts as a click-through.

AB Picturer Dashboard

The AB Picturer Dashboard shows you the CTR and other statistics for each image. The image with the higher CTR wins and can be used as the main image for the post. It is a very simple dashboard at the moment, but I plan to improve it in the future.

I've set up a Discord for you to try out the AB Picturer and give me feedback. I'm super curious about your feedback as I plan to develop more features based on your needs.

Also, I think it is pretty cool to be part of a community. So join the AB Picturer Discord and let's have some fun together.

And please subscribe to our Newsletter if you want regular updates about AB Picturer.

In the next section I will show you how to integrate and use AB Picturer.

Integrate AB Picturer into your blog

It doesn't matter what blog post engine you're using, as long as it supports displaying images, it should work.

Gastby and AB Picturer

So on my blog post I have the two urls preview and visitor in the meta section:

imagePreviewUrl: "https://api.ab-picturer.com?projectId=sst&state=preview"
imageVisitorUrl: "https://api.ab-picturer.com?projectId=sst&state=visitor"

I then simply render these to the thumbnail, meta tags and post itself:

Here is the snippet to add it as a thumbnail:

const { imagePreviewUrl } = node.frontmatter
...
<img src={imagePreviewUrl} ... />
...

See the complete example in my repository

Add it to the meta tags:

<MetaTags
 title={title}
 description={props.data.markdownRemark.excerpt}
 thumbnail={(thumbnail && url + thumbnail) || imagePreviewUrl}
 url={url}
 pathname={props.location.pathname}
/>

And finally, add it to the post

{
 imageVisitorUrl && <img src={imageVisitorUrl} alt="Title" /> }
}

This is cool, right? It is basically just a src url :)!

Outlook

I'm a big fan of the blog post site [dev.to] (https://dev.to). I tried to get the AB pictures to work there but unfortunately I failed. I think they do some caching for the images. I'll try to contact them and see if we can get it to work.

I could collect more information like where the visitor is coming from or what country the user is from. If you would like to have this and other features, contact me in our AB Pictures Discord :)!

Conclusion

Rating your images to see if your audience likes them is super important. In this article I have shown you how to do it with AB Picturer. Join the AB Picturer Discord.

Create a Next.js Server Component S3 Picture Uploader with SST

Wed, 03 Jan 2024 00:00:00 GMT

I recently started exploring SST as an alternative to my favorite full-stack set consisting of Projen, AWS CDK, and React. I have been thoroughly impressed with the experience so far. In this article, I will demonstrate how to create a Next.js App Router S3 Picture Uploader using SST.

SST

SST is a powerful framework that simplifies the development of serverless applications. It offers a straightforward and opinionated approach to defining serverless apps using TypeScript. Built on top of AWS CDK, SST handles the complexity of setting up your serverless infrastructure automatically. SST is an open-source framework and is completely free to use.

SST offers a variety of powerful constructs, including the NextjsSite construct. In the following section, I will provide more details about the NextjsSite construct, which greatly simplifies the process of deploying your frontend.

NextjsSite

The NextjsSite construct allows you to effortlessly create and manage open-next, which is a great alternative for hosting Next.js on Vercel. Being defined within an SST App, you can easily integrate other AWS services, making it incredibly powerful. However, if you're using the latest and most advanced features of Next.js, such as Server Components with the Next.js App Router, you may encounter some challenges.

Server Components

Server Components are a new way to build with Next.js. They allow you to write parts of your application using React components that are served from the server. This approach offers several advantages, such as faster page loading and simplified setup without the need to manage client states with useState, useEffect, and similar hooks. However, working with Server Components may require adjusting your workflow and learning new concepts.

S3 Picture Uploader

In this section, I will demonstrate how to create an S3 picture uploader using the Next.js App Router and SST. We will utilize the NextjsSite construct to create the Next.js App Router and the S3 construct to create the S3 Bucket.

You can find all the code in my GitHub Repository.

Initialize SST NextjsSite

All the steps are taken from the official SST guide.

npx create-next-app@latest

Mainly choose the defaults. Then switch to the app folder and open it via VS Code.

cd <SST_PROJECT>
code .

Now run:

npx create-sst@latest
npm install

Before deploying to your AWS Account, ensure that you have set up the correct credentials. I recommend using the AWS Identity Service to obtain temporary AWS CLI credentials, but you can also set up IAM User credentials or profiles. Once you have the credentials in place, run the following command:

npx sst deploy

Ensure that the AWS deployment is successful! View your CloudFront SiteUrl shown in the SST Output. Voila! You now have a running Next.js application on AWS with open-next 🤯. Let's take a closer look at what has been deployed in our AWS account because it's quite extensive!

To inspect the resources created by SST, go to the AWS Console and navigate to CloudFormation. Click on the newly created stack to view the details. You will find a set of helper Lambda Functions, the Lambda Function and Lambda URL for the Server Component, a CloudFront Distribution, and an S3 bucket that serves the static Next.js files.

Add S3 Picture Uploader

Ok let's go! We need to add an S3 bucket where we can upload the pictures to. Go to the sst.config.ts file and add a Bucket:

import { SSTConfig } from "sst"
import { Bucket, NextjsSite } from "sst/constructs"

export default {
 config(_input) {
  return {
   name: "sst-nextjs-s3-picture-uploader",
   region: "us-east-1",
  }
 },
 stacks(app) {
  app.stack(function Site({ stack }) {
   const bucket = new Bucket(stack, "public")
   const site = new NextjsSite(stack, "site", {
    permissions: [bucket],
    bind: [bucket],
   })

   stack.addOutputs({
    SiteUrl: site.url,
   })
  })
 },
} satisfies SSTConfig

The permissions: [bucket] property gives the NextjsSite read and write access for the S3 bucket. With bind: [bucket] You can use SST node variables in your React code like Bucket.public.bucketName. Let's update the page.tsx for adding the upload button and S3 AWS SDK upload code. You might need to run npm run dev so that the SST node variable Bucket.public.bucketName gets recognized.

import { Bucket } from "sst/node/bucket"
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3"

export default async function Home() {
 async function upload(data: FormData) {
  "use server"

  const file: File | null = data.get("file") as unknown as File
  if (!file) {
   throw new Error("No file uploaded")
  }

  const bytes = await file.arrayBuffer()
  const buffer = Buffer.from(bytes)

  const client = new S3Client({ region: "us-east-1" })

  const command = new PutObjectCommand({
   Bucket: Bucket.public.bucketName,
   Key: file.name,
   Body: buffer,
   ACL: "public-read",
  })

  await client.send(command)

  console.log(`Uploaded ${file.name} to S3`)

  return { success: true }
 }

 return (
  <main className="flex min-h-screen flex-col items-center justify-between p-24">
   <form action={upload}>
    <input name="file" type="file" accept="image/png, image/jpeg" />
    <button type="submit">Upload</button>
   </form>
  </main>
 )
}

Do you see how seaminess we can sneak in the AWS SDK s3 upload code? For me, that is mind-blowing if you like to compare it with a client-side variant where you couldn't do that so easily without exposing your AWS API credentials. But as server components are server side we are saved. It offloads a lot of complexity. Super cool!

Let's deploy that! For more convenience let's add a new command in the package json "deploy": "sst deploy",. Now run:

npm run deploy

Open the CloudFront SiteUrl. Now click on the upload button and check if you can see the file in S3.

BTW. for faster development you could also run locally with npm run dev but make sure to load your AWS CLI credentials before which allowing access to the S3 Bucket.

Conclusion

I'm still super flashed at how nicely SST is orchestrating the frontend with the backend. In this post, I described how you can start with SST and how to create an S3 picture uploader. I hope you learned something new. If you liked my post or want to correct me please reach out to me :).

I am passionate about contributing to Open Source projects. You can find many of my projects on GitHub that you can already benefit from.

And don't forget to visit my site

How to Perform Unit Testing on your AWS Bedrock AI Lambda

Sat, 30 Dec 2023 00:00:00 GMT

Using the AWS Bedrock API for MVPs is incredibly enjoyable! I recently wrote an article on how you can make the LLM Claude respond in JSON. You can check it out here. While it's a lot of fun, testing your LLM settings and prompts can become tiresome and frustrating. In this article, I will explain how you can effectively unit test your Lambda function that calls the AWS Bedrock API. By being able to unit test your prompts, you can iterate quickly towards your desired MVP or project state.

Lambda Unit Testing

Unit testing a Lambda function is straightforward since it is essentially a function that executes code. To test it, you can simply call the function with the necessary arguments and verify the response. If you are using Node.js, Jest provides a convenient way to run your test code.

Lambda Streaming Response & Unit Testing

The Lambda Streaming Response allows to leverage the streaming response from your LLM. Using the streaming response from your LLM is important as it gives your early feedback. That is pretty much what is happening when ChatGPT gives your those word by word streaming response.

Lambda

Here I show you a bit from the Lambda Function I want to unit test later.

async function handler(
  event: APIGatewayProxyEventV2WithRequestContext<APIGatewayEventRequestContextV2>,
  responseStream: lambdaStream.ResponseStream,
  ctx?: Context,
) {
  console.log(`event: ${JSON.stringify(event)}`);

  const body = event.body ? JSON.parse(event.body) : undefined;
  const { userInput, ninoxTables } = body;

  ...

  const bedrockParams: InvokeModelCommandInput = {
    modelId: 'anthropic.claude-v2:1',
    contentType: 'application/json',
    accept: '*/*',
    body: JSON.stringify({
      prompt: `\n\nHuman: ${prompt}\n\nAssistant:`,
      temperature: 0,
      ...
    }),
  };
  console.log(`bedrockParams: ${JSON.stringify(bedrockParams)}`);

  const command = new InvokeModelWithResponseStreamCommand(bedrockParams);

  //InvokeModelWithResponseStreamCommandOutput
  const response: any = await client.send(command);
  ...

  responseStream.end();
}

Unit Testing

And here is the unit test in a file living next to my Lambda function.

import { ArcbotLambdaInput, handler } from '../src/arcbot-stack.stream';

...

test('userInput: How is the weather?', async () => {
  const response = await handler(
    mockEvent({
      userInput: 'How is the weather?',
    }),
    //@ts-ignore
    '',
  );
  console.log(response);

  // do some validation
  expect(response.statusCode).toEqual(200);
  expect(JSON.parse(response.body)).toEqual({
    respond: 'I do not understand. Please rephrase!',
  });
});

test('Create a customer table', async () => {
  const response = await handler(
    mockEvent({
      userInput: 'Create a customer table',
    }),
    //@ts-ignore
    '',
  );

  // do some validation
  commonExpects(response);
  const body = JSON.parse(response.body) as Record<string, NinoxTable>;
  console.log(`body: ${JSON.stringify(body, null, 2)}`);

  const validationResult = z.record(NinoxTableSchema).safeParse(body);
  if (!validationResult.success) {
    console.log(validationResult.error.message);
  }
  expect(validationResult.success).toBeTruthy();

  expect(Object.entries(body)[0][0]).toBe('customer');
  expect(Object.entries(body)[0][1].caption).toBe('Customer');
});

const mockEvent = ({ userInput }: ArcbotLambdaInput) => {
  const event: lambda.APIGatewayProxyEventV2 = {
    version: '',
    ...
    body: JSON.stringify({ userInput } as ArcbotLambdaInput),
  };

  return event;
};

Conclusion

Unit testing your Lambda function that calls the AWS Bedrock API is crucial. It enables you to iterate quickly towards your desired MVP or project state. I hope this article has been helpful to you. If you have any questions or feedback, please don't hesitate to reach out to me.

I am passionate about contributing to Open Source projects. You can find many of my projects on GitHub that you can already benefit from.

And don't forget to visit my site

My first Experience with Powertools for AWS Lambda (TypeScript)

Sun, 24 Dec 2023 00:00:00 GMT

Hi there,

I'm new to Powertools for AWS Lambda (TypeScript). Though I hear a lot of my DevOps friends say that it's a great tool for building serverless applications, I've never had the chance to use it myself. So I decided to give it a try and see what all the fuss is about. What can I say, I totally love it. In the next sections I will explain what the.

What is Powertools for AWS Lambda

Powertools for AWS Lambda is a collection of utilities, patterns, and best practices for writing AWS Lambda functions in Python, Typescript, Java and DotNet. It includes logging, tracing, custom metrics, and more. The goal of this project is to enable developers to build scalable and robust serverless applications easily.

Why I am using AWS Lambda Powertools

I'm a huge DevOps fanboy so I'm all over for techniques like Infrastructure as Code (IaC) or Serverless. Pretty cool about those that it helps me to focus on my business logic of my software product rather then wasting time in setting it up. Powertools feels pretty similar as it helps me to easy implement certain goodies as logging or tracing without the need to spend much time.

Certainly my main motivation is the Powertools logger library. The logger library helps you to write my Lambda logs in specific format to enable features like log levels and log queries with AWS CloudWatch Logs Insights. So far that is amazingly great and it gives me a powerful insight into my Lambda.

I'm using Powertools for me newest AI MVP. Part of the MVP is a Lambda where I call the AWS Bedrock API. Furthermore I automatically validate the LLM response. For more details see https://martinmueller.dev/aws-bedrock-validation.

How to start?

The Powertools GitHub repository offers nice examples to integrate and learn from. For example here are nice example written in AWS CDK. Following I will list the features I gained some experience with.

Logger

The logger is super fun to use! It gives me a powerful insight about my Lambda functions. Though I'm still not super sure when to use the different log levels and when and what objects I should put into the object logger part. But I will keep learning from my DevOps friends and make my own experience. Ultimately I will know better when I truly need insight via Logs Insights.

Parameters

The parameters feature is super cool. It helps me to easily access the parameters I defined in my AWS CDK stacks. I can access them via the get function. So far I only used it for getting secrets from the secrets manager but I'm super excited to use it for more like perhaps app configurations.

Idempotency

The idempotency feature seems super useful. It could help to reduce the quite high costs of using the AWS Bedrock API. But unfortunately idempotency isn't currently combinable with Lambda stream response. Hopefully that will change in future 🤞.

Resources to learn utilizing Powertools

Here's a list of examples I found helpful to learn how to apply Powertools:

Examples from the GitHub Repo:

https://github.com/aws-powertools/powertools-lambda-typescript

Powertools has ana amazing Discord community. Make sure to not miss it!:

Discord

Lee Gilmore:

https://github.com/leegilmorecode/embedded-aws-cloudwatch-dashboards/tree/main

Conclusion

Working with AWS Lambda Powertools Typescript totally make sense and I love it. It will definitely be my default choice when developing my next Lambda. I still need to learn how to use it properly! Please if you have any feedback how I can utilize Powertools better, reach out to me!

I am passionate about contributing to Open Source projects. You can find many of my projects on GitHub that you can already benefit from.

And don't forget to visit my site

Automatically validate your AWS Bedrock LLM Responses

Mon, 18 Dec 2023 00:00:00 GMT

Validating the response from your Language Learning Model (LLM) is a critical step in the development process. It ensures that the response is in the correct format and contains the expected data. Manual evaluation can quickly become tiresome, especially when making frequent changes to your LLM. Automating or partially automating the validation process is highly recommended to save time and effort. In this post, I will discuss and demonstrate some ideas how you can achieve this automation.

Before

When I refer to LLM, I am talking about the use of existing foundation models through AWS Bedrock, such as Claude, LLama2, and others. You can learn more about AWS Bedrock here. There are techniques you can use to enhance the response, such as prompt refinements, RAG (Retrieval Augmentation using Vector Databases), or fine-tuning.

Responses from Language Learning Models (LLMs) are often non-deterministic, meaning that different responses can be generated even with the same prompt. However, this behavior can be adjusted to some extent using LLM parameters such as temperature.

Ideas

In the following sections I will present some ideas for creating automated tests for your LLM responses. I will also provide some examples of how I implemented these ideas in my own projects.

Validate the Shape

In many cases, the response may contain deterministic parts that can be used to partially validate it. For instance, I rely on Claude to provide a JSON response. I have taught Claude the schema of the JSON response, and by performing a schema validation test, I can verify if Claude adheres to the schema. Verifying a JSON schema is very simple.

Each programming language has a library that can be used to validate the schema. For instance, in TypeScript, I use the zod library to create and validate the schema. Which looks like that:

import { z } from 'zod';

export const NinoxFieldSchema = z.strictObject({
  base: z
    .enum([
      'string',
      'boolean',
      ...
    ])
    .optional(),
  caption: z.string().optional(),
  captions: z.record(z.string()).optional(),
  required: z.boolean().optional(),
  order: z.number().optional(),
  ...
});

export type NinoxField = z.infer<typeof NinoxFieldSchema>;

export const NinoxTableSchema = z.strictObject({
  nextFieldId: z.number().optional(),
  caption: z.string().optional(),
  captions: z.record(z.string()).optional(),
  hidden: z.boolean().optional(),
  ...
});

export type NinoxTable = z.infer<typeof NinoxTableSchema>;

And as part of my unit tests:

test('check schema', async () => {
    ...

    const body = JSON.parse(response.body);

    const validationResult = NinoxTableSchema.safeParse(
        JSON.parse(body.json),
    );
    if (!validationResult.success) {
        console.log(validationResult.error.message);
    }
    expect(validationResult.success).toBeTruthy();
});

Validate Sub-Responses

In my current AI application, I utilize multiple LLM calls to generate the final response. While validating the entire response may be challenging, I can easily validate some of the sub-responses. For instance, I have a deterministic response for which I can verify the response. The deterministic response classifies a user's intent into a specific category. For example, if the user asks to create a table, the intent is classified as "create_table". That will generate a deterministic sub-response in my AWS Lambda for the "create_table" intent. To test the accuracy of the classification, you can use well-known methods such as train-validation-test Split the training data into subsets for training and validation. I'll describe this technique more in the next section.

Train-Validation-Test Split

The train-validation-test split is very crucial to messsure the performance of the LLM. One methods of those splits is the k-fold cross validation. I try to explain this approach in easy words. Make sure to check the far more technical artical from Everton Gomede, PhD, The Significance of Train-Validation-Test Split in Machine Learning!

For instance, you could use 90 percent of the data for training and 10 percent for validation. Then, you can use the validation data to test the accuracy of the classification. Additionally, you can use permutation to shift the 10 percent of the validation data. I implemented a simple algorithm in TypeScript which helps me to calculate the accuracy of the classification:

import { test } from "@jest/globals"
import * as ArcbotStackStream from "../src/arcbot-stack.stream"
import {
 call_bedrock,
 generate_intent_identification_prompt,
 generate_table_identification_prompt,
 modify_table_prompt,
 relationship_json_prompt,
} from "../src/arcbot-stack.stream"
import {
 intentTrainingsData,
 modifyTableTrainingsData,
 oneToManyTrainingsData,
 tableIdentificationData,
} from "../src/training-data"

const runEvaluation = async <T extends { [s: string]: string[] }>(
 trainingData: T,
 jestSpy: jest.SpyInstance<T, [], any>,
 promptRefinement: (userInput: string) => Promise<string>,
 jsonResponse?: boolean
) => {
 const getTrainingAndEvaluationPermutations = (trainingsData: T) => {
  // Split trainings data into training and evaluation data
  const sliceTrainingsData = (fromPercentage: number, toPercentage: number) =>
   Object.entries(trainingsData).reduce(
    (acc, data) => {
     const evaluationSlice = data[1].slice(
      data[1].length * fromPercentage,
      data[1].length * toPercentage
     )
     const trainingSlice = data[1].filter((d) => !evaluationSlice.includes(d))
     return {
      training: { ...acc.training, [data[0]]: trainingSlice } as T,
      evaluations: {
       ...acc.evaluations,
       [data[0]]: evaluationSlice,
      } as T,
     }
    },
    {
     training: {} as T,
     evaluations: {} as T,
    }
   )
  const trainingPercentage = 0.9
  const validationPercentage = 1 - trainingPercentage

  // permute the training and evaluation data
  const trainingValidationPermutations = [
   sliceTrainingsData(0, 0 + validationPercentage),
  ]
  for (let i = 0 + validationPercentage; i < 1; i = i + validationPercentage) {
   trainingValidationPermutations.push(
    sliceTrainingsData(i, i + validationPercentage)
   )
  }

  console.log(
   `trainingValidationPermutations: ${JSON.stringify(
    trainingValidationPermutations
   )}`
  )
  return trainingValidationPermutations
 }

 let correctResponses = 0
 let wrongResponses = 0

 const trainingRecords = getTrainingAndEvaluationPermutations(trainingData)

 for (const trainingPermutation of trainingRecords) {
  console.log(`trainingPermutation=${JSON.stringify(trainingPermutation)}`)

  jestSpy.mockImplementation(() => trainingPermutation.training)

  for (const evaluationRecords of Object.entries(
   trainingPermutation.evaluations
  )) {
   for (const input of evaluationRecords[1]) {
    const intent_prompt = await promptRefinement(input)
    const response = await call_bedrock(intent_prompt, jsonResponse)

    let received = response

    // trim to JSON string
    if (jsonResponse) {
     received = JSON.stringify(JSON.parse(received))
    }

    console.log(`Expected ${evaluationRecords[0]}\nReceived ${received}`)

    if (evaluationRecords[0] === received) {
     correctResponses++
    } else {
     wrongResponses++
    }
   }
  }
 }
 console.log(
  ` correctResponses: ${correctResponses}\n wrongResponses: ${wrongResponses} \n ${
   correctResponses / (correctResponses + wrongResponses)
  } accuracy`
 )
}

test("evaluate one to many", async () => {
 const mockTrainingsData = jest.spyOn(
  ArcbotStackStream,
  "getOneToManyTrainingData"
 )

 const oneToManyPromptRefinement = async (userInput: string) =>
  relationship_json_prompt(
   {
    CUSTOMER: { caption: "Customer" },
    EMPLOYEE: { caption: "Employee" },
    INVOICE: { caption: "Invoice" },
   },
   userInput
  )

 await runEvaluation(
  oneToManyTrainingsData,
  mockTrainingsData,
  oneToManyPromptRefinement,
  true
 )
})

test("evaluate intent", async () => {
 const mockTrainingsData = jest.spyOn(
  ArcbotStackStream,
  "getIntentTrainingData"
 )

 const generate_intent_identification_promptRefinement = async (
  userInput: string
 ) => generate_intent_identification_prompt(userInput)

 await runEvaluation(
  intentTrainingsData,
  mockTrainingsData,
  generate_intent_identification_promptRefinement
 )
})

test("evaluate table identification", async () => {
 const mockTrainingsData = jest.spyOn(
  ArcbotStackStream,
  "getTableIdentificationData"
 )

 const generate_table_identification_promptRefinement = async (
  userInput: string
 ) => {
  const prompt = generate_table_identification_prompt(
   userInput,
   Object.keys(tableIdentificationData)
  )

  return prompt
 }

 await runEvaluation(
  tableIdentificationData,
  mockTrainingsData,
  generate_table_identification_promptRefinement
 )
})

test("evaluate modify table", async () => {
 const mockTrainingsData = jest.spyOn(
  ArcbotStackStream,
  "getModifyTableTrainingsData"
 )

 modifyTableTrainingsData

 const modify_table_promptRefinement = async (userInput: string) => {
  const prompt = await modify_table_prompt({}, userInput)

  return prompt
 }

 await runEvaluation(
  modifyTableTrainingsData,
  mockTrainingsData,
  modify_table_promptRefinement,
  true
 )
})

I think the most interesting part here is the method interface getTrainingAndEvaluationPermutations(trainingData) as that always expect the same format as input and gives you back a permuted test validation split of the input training data. The training data have to be in record string list shape:

<T extends { [s: string]: string[] }>

Where the key represents the expected result / classification class / LLM output and the value represents possible inputs which leads to the result. The result will be this type:

{
 training: T
 evaluations: T
}
;[]

It is an Array representing the permutations. Each permutation has a training and evaluations slice.

One traing data example would be:

export const intentTrainingsData: { [key: string]: string[] } = {
 create_new_table: [
  "Create table to store invoices",
  "I need to store my customers information",
  "I need a table for my employees",
 ],
 modify_existing_table: [
  "Customers table should also have an address",
  "Add address to the customer table",
  "Invoice should have a date",
 ],
 link_two_tables: [
  "Customer should have multiple invoices",
  "Each employee should be responsible for multiple customers",
 ],
 do_not_know: [
  "How are you today?",
  "What is your name?",
  "What is the weather today?",
 ],
}

This training set is to teach the model the intent recognition of the user. The permuted training validation would looks like:

[
 {
  training: {
   create_new_table: [
    "Create table to store invoices",
    "I need to store my customers information",
   ],
   modify_existing_table: [
    "Customers table should also have an address",
    "Add address to the customer table",
   ],
   link_two_tables: [
    "Customer should have multiple invoices",
   ],
   do_not_know: [
    "How are you today?",
    "What is your name?",
    "What is the weather today?",
    "2 + 3",
   ],
  },
  evaluations: {
   create_new_table: [
     "I need a table for my employees",
   ],
   modify_existing_table: [
     "Invoice should have a date",
   ],
   link_two_tables: [
    "Customer should have multiple invoices",
    "Each employee should be responsible for multiple customers",
   ],
   do_not_know: [
    "How are you today?",
    "What is your name?",
   ],
  },
 },
 {
  training: {
   create_new_table: [
    "Create table to store invoices",
    "I need to store my customers information",
    "I need a table for my employees",
   ],
   modify_existing_table: [
    "Customers table should also have an address",
    "Add address to the customer table",
    "Invoice should have a date",
   ],
   link_two_tables: [
    "Each employee should be responsible for multiple customers",
   ],
   do_not_know: [
    "How are you today?",
    "What is your name?",
    "What is the weather today?",
    "2 + 3",
   ],
  },
  evaluations: {
   create_new_table: [
    "Create table to store invoices",
    "I need to store my customers information",
    "I need a table for my employees",
   ],
   modify_existing_table: [
    "Customers table should also have an address",
    "Add address to the customer table",
    "Invoice should have a date",
   ],
   link_two_tables: [
    "Customer should have multiple invoices",
    "Each employee should be responsible for multiple customers",
   ],
   do_not_know: [
    "What is the weather today?",
   ],
  },
 },
]

Golden Response

This is an idea from the AI community that shows promise. Although I haven't personally tested it yet, the concept is to compare the response with a "golden response" to ensure its correctness. The golden response can be compared with the actual response with the same Language Learning Model (LLM). We can then determine if they are identical or very similar. This approach holds potential and I'm eager to try it out soon.

Thanks

I would like to express my gratitude to the AWS Community for their invaluable assistance in helping me.

A special thanks goes to Chris Miller for giving me a lot of thoughts and feedback on my validation approach. Neylson Crepalde for making me aware and explaining the golden response validation method.

Once again, thank you all for your support and contributions.

Conclusion

Working with AWS Bedrock AI is incredibly enjoyable. The field is constantly evolving, and there is always something new to learn. In this post, I explained how to partly validate your LLM responses.

I am passionate about contributing to Open Source projects. You can find many of my projects on GitHub that you can already benefit from.

And don't forget to visit my site

AWS Bedrock Claude 2.1 - Return only JSON

Thu, 07 Dec 2023 00:00:00 GMT

Working with the AWS Bedrock API is an exhilarating experience! I came across an interesting business case where I needed to develop an AI MVP. The MVP generates JSON data based on a prompt and utilizes the anthropic.claude-v2:1 model in AWS Bedrock.

I encountered an issue where the response I received was not pure JSON. It contained additional characters that I couldn't remove like:

" format: {\"one\":\"Supplier\",\"many\":\"Time\"}"

Seeking help from the AWS Community, I was able to find a solution to this problem. In this post, I will share the solution with you.

The Problem

Human: $YOUR_PROMPT . Answer in JSON formatAssistant:{,

This technique is known as "Put words in Claude’s mouth". It involves providing a prompt to Claude and letting it generate the rest of the response on its own. While there may be alternative approaches to solving this issue, I am currently happy with this solution.

Thanks

I would like to express my gratitude to the AWS Community for their invaluable assistance in helping me resolve this issue.

A special thanks goes to Corvus Lee for providing the advice that ultimately solved the problem.

I would also like to thank Ken Collins for bringing the Claude 2 docs sheet to my attention.

Once again, thank you all for your support and contributions.

Conclusion

Working with AWS Bedrock AI is incredibly enjoyable. The field is constantly evolving, and there is always something new to learn. In this post, I demonstrated how to obtain a pure JSON response from AWS Bedrock Claude 2.1. As AI technology continues to advance rapidly, you may not encounter this issue in the future. However, if you are working with Claude 2.1 or newer, be sure to refer to the documentation for more information.

I hope you found this post helpful, and I look forward to sharing more with you in the future.

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on the:

And don't forget to visit my site

Java Spring CI/CD in AWS - The HalloCasa Journey

Mon, 28 Aug 2023 00:00:00 GMT

A fresh start can be daunting, especially when it comes to revamping an existing Java Spring application. This was my journey with the HalloCasa real estate platform, and here's how we took it from deployment nightmares to smooth, cloud-native operations. If you would like to improve your Java Spring CI/CD experience as well. Reach out to me!

As well I have a recording about the Java Spring CI/CD experience on YouTube. Check it out here

Introducing HalloCasa

HalloCasa is a dynamic Real Estate Platform designed to allow real estate agents to create their own business card within a couple of minutes. It also serves as a directory and a referral network.

My motivations to work on HalloCasa are that I had the chance to contribute to the development of a powerful product. My expertise in AWS could be applied effectively. Working with a co-founder who was not only business-savvy but also had a knack for generating compelling content in the real estate area. He has his own podcast https://blog.hallocasa.com/podcasts/. Make sure to check it out!

The Deployment Dilemma

In the beginning, our deployment process could only be described as patchwork:

Setting up Maven, Tomcat, and MySQL locally. Creating a war file and then SSHing into an EC2 VM to transfer it. Manually restarting the Tomcat server each time.

Our NextJS frontend wasn't much different. We had to SSH into our EC2 VM and manually run “npm start”. And to make matters a bit more convoluted, our frontend was in BitBucket while the backend resided in AWS CodeCommit.

Deploying the New and Improved Way

Here’s how I revamped our deployment:

Shifting to GitHub

Migrated from AWS CodeCommit and BitBucket to GitHub. Leveraged GitHub Actions for test builds and introduced a PR AI reviewer for efficiency.

Dockerization

Deployed our configuration as code, which is both clean and manageable. With docker-compose, local development became hassle-free. Gone were the days of setting up Maven, Tomcat, and MySQL locally. Further utilized this to feed AWS ECS, making deployments consistent. Additionally health checks were introduced to ensure that the application is up and running. If the health check fails, the application will restart and in most cases that fixes the problem.

CDK Pipelines

Implemented a CI/CD staging pipeline that first deploys changes onto QA and subsequently to PROD. Ensured robustness by integrating testing with Postman. We setup a dedicated AWS account for each stage, ensuring clear demarcation and management.

DB Migration with Flyway

Adopted Flyway to empower our Java App to manage the database, eliminating the need to provide access to the database for schema modifications.

Monitoring Metrics

Introduced plenty of metrics to monitor our backend. For an informed decision on crucial metrics, I sought advice from ChatGPT about the most essential metrics for a Java application. Our revamped production setup went live on 01/08/23, and the migration was seamless.

Reflecting on the Journey

The refurbishment of HalloCasa reaffirmed several beliefs:

Docker is a lifesaver. The portability and consistency it brings to applications are unparalleled. CI/CD is a game-changer. It automates manual tasks and ensures faster, reliable deliveries. AWS CDK is powerful. It simplifies cloud resource provisioning and management. CDK Pipelines streamline deployments. With it, managing multiple stages of deployment becomes a breeze. One revelation that stood out was how the lines between infrastructure and application seemed blurred in our setup. And this convergence was a positive one, indicating tight integration and consistency.

The HalloCasa journey is a testament to the power of continuous learning, adaptation, and innovation. Our users now enjoy a more robust platform, and we can sleep better at night, knowing that our deployment is smoother than ever. If you have a Java application which needs a bit polishing, reach out to me!

I hope you enjoyed this post and I look forward to seeing you in the next one.

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on the:

And don't forget to visit my site

Create your Prototype or MVP with the T3 Stack

Wed, 05 Apr 2023 00:00:00 GMT

What is a Prototype?

A prototype is a preliminary version of a product, often used for testing and validation purposes. It allows you to experiment with different ideas and get feedback from users before investing significant time and resources into development.

What is an MVP?

An MVP, or Minimum Viable Product, is a more developed version of a prototype that includes only the essential features needed to validate the concept and get it into the hands of early adopters. It is the smallest possible version of your product that can still provide value to your users. It allows you to test your assumptions and get feedback from real users, which you can then use to improve and refine your product.

Why should I develop a Prototype or MVP?

Developing a Prototype or MVP can help you save time and money in the long run. By validating your ideas early on and getting feedback from users, you can avoid costly mistakes and ensure that you are building something people actually want. It also allows you to test your assumptions and make data-driven decisions, which can increase the chances of success for your product.

How can I develop a Prototype or MVP?

There are several ways to develop a Prototype or MVP like No or Low-Code solutions. But one effective tool is the T3 Stack. This stack combines NextJS, TailwindCSS, tRPC, and Prisma to create a powerful and efficient development environment. NextJS provides server-side rendering and easy routing, while TailwindCSS offers a library of pre-built UI components. tRPC is a simple and lightweight framework for building remote procedure call APIs, and Prisma provides a type-safe database client. Together, these technologies allow you to quickly develop and deploy your Prototype or MVP with confidence.

What is the advantage of using the T3 Stack compared to No or Low-Code solutions?

The advantage of using the T3 Stack over No or Low-Code solutions is that it provides more flexibility and control over the development process. While No or Low-Code solutions may be easier to use, they can be limiting in terms of customization and scalability. The T3 Stack, on the other hand, allows for more complex and customizable development while still being relatively easy to use, making it a great option for developing a Prototype or MVP.

Outlook

In one of my next posts, I give you deeper insight into the T3 Stack like how it works. I also show you how to use it to develop a Prototype or MVP.

A huge thanks to the T3 Stack community for their hard work and support. And a special thanks to Theo for creating this amazing stack and thriving it. I love your content on YouTube. Keep up the amazing work!

I hope you enjoyed this post and I look forward to seeing you in the next one.

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on the:

And don't forget to visit my site

Chat Indexing with Chatsindex.com

Tue, 21 Mar 2023 00:00:00 GMT

Would you like to grow your chat community? Chatsindex.com indexes chat messages and make them searchable for search engines like Google. Well, will be as this is still a prototype! Through SEO (Search Engine Optimization) it is ensured that the entries appear as high as possible in the search results. This way new users can be attracted to your Discord server or any other chat you prefer.

Below you can see an example of how a Discord channel and the search results can look like:

The Google page is only mocked (Chatsindex.com/search) because it is still a prototype.

How does it work?

A Discord bot writes the messages into a database. The database gets queried by a NextJS server and rendered into search-engine-optimized (SEO) pages. Google crawls the page and the entries are discoverable.

The prototype is built using the T3-Stack. The T3-stack is a NextJS stack with TailwindCSS, tRPC, and Prisma. The database is currently SQLite and is queried with tRPC and Prisma. The page is rendered using NextJS and TailwindCSS.

Participants

Does this idea sound interesting to you? Then feel free to contact me https://MartinMueller.dev. I am looking for volunteers with Discord servers to test the idea.

Outlook

If the idea is well received, I would like to index other private chat systems like WhatsApp groups, Telegram groups, Slack, etc.

Another idea of mine is to implement an overall search for private chats like Discord, WhatsApp groups, Telegram groups, and Slack. With certain keywords or categories, private chats with certain topic content can then be found. For example, if I'm interested in financial education, I can then simply enter finance in the search and I'll be presented with Discord servers, WhatsApp/Telegram groups, etc. with a focus on finance. If you like the idea, feel free to contact me.

Conclusion

I think the combination of SEO and private chats like Discord servers is an exciting idea. For now, I want to test the idea and see if there is any demand for it. If you are interested, feel free to contact me.

Using the T3-Stack was super fun and it took me just like a weekend to sketch out this little prototype. If you need help with your idea or project reach out to me as well!

Thanks to the DeepL translater (free version) for helping with translating to English and saving me tons of time :).

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on the:

And don't forget to visit my site

Amplify AppSync Permission

Sun, 05 Feb 2023 00:00:00 GMT

Hi,

With Amplify AppSync it is possible to configure an AWS AppSync with declarative code. With Directives like @model, @auth or @function other AWS services like DynamoDB, Cognito or Lambda can be connected to AppSync in a meaningful way. This makes it a powerful tool. With just a few lines of GraphQL code, a lot can be configured in AWS, so to speak.

Below you can see an Amplify AppSync example:

type User
  @model
  @key(fields: ["email"])
  @auth(
    rules: [
      { allow: groups, groups: ["admin"] }
      { allow: private, provider: iam }
      { allow: owner, ownerField: "email", identityClaim: "email" }
    ]
  ) {
  email: String!
  role: Role!
  ...
}

There is a lot going on here! With the @model directive, a DynamoDB table with the name User is automatically created which stores the user entries as DynamoDB items. With @key the email is set as PK (Partition Key). The permission, who is allowed to access the items, is defined with @auth. A user in the admin group is allowed to query, mutate and subscribe to all items.

With { allow: private, provider: iam } an IAM entity that has the required permissions is also allowed to querry, mutate and subscribe to the items. I use this for example with lambdas that should perform certain tasks like creating new users in the table when they log in via Cognito for the first time.

Last but not least with { allow: owner, ownerField: "email", identityClaim: "email" } Cognito users who have identified themselves via oauth2 are allowed to access their items. The identity claim email is supplied by the Cognito JWT token and is then matched with the email defined in the item.

Pretty cool, isn't it? With just these few lines of Amplify AppSync GraphQL code, we get a lot of functionality.

Permission for shared items

Defining permission with @auth works very well for simple cases like the user example shown here. But what if you want to access items from multiple users? Well here are several possibilities I want to introduce. But first I will introduce the table with the shared items:

type Project
  @model
  @auth(
    rules: [
      { allow: groups, groups: [ "admin"] }
      { allow: private, provider: iam }
    ]
  ){
  id: ID!
  projectNumber: Int!
  ...
}

A project should be accessible to several users. The simplest approach would be to store the users in a string list.

String-List

The code must be adapted only minimally with:

type Project
  @model
  @auth(
    rules: [
      { allow: groups, groups: [ "admin"] }
      { allow: private, provider: iam }
      { allow: owner, ownerField: "userEmails", identityClaim: "email" }
    ]
  ){
  id: ID!
  projectNumber: Int!
  userEmails: [String] # <-- user string list
  ...
}

The userEmails string list now just needs to be kept up to date with the emails from the users who can access it. However, managing the userEmails is costly and requires additional code and space in the item.

Virtual Lambda

We could also use a virtual lambda which dynamically calculates if the user should have access or not.

type Project
  @model
  @auth(
    rules: [
      { allow: groups, groups: [ "admin"] }
      { allow: private, provider: iam }
      { allow: owner, ownerField: "userEmail", identityClaim: "email" }
    ]
  ){
  id: ID!
  projectNumber: Int!
  userEmail: String @function("userEmailLambda")
  ...
}

The solution is impressive because of the reduced space in the DynamoDB table. However, with a large number of items it would mean a large number of lambda calls which would drive up the cost. Also, the delay due to the lambda call can be too significant.

JWT Claim

This idea is probably the most creative. Using the pre-token-generation trigger Lambda a claim can be set which signals which item may be accessed. The code for Amplify AppSync looks something like this:

type Project
  @model
  @auth(
    rules: [
      { allow: groups, groups: [ "admin"] }
      { allow: private, provider: iam }
      { allow: owner, ownerField: "id", identityClaim: "currentProjectId" }
    ]
  ){
  id: ID!
  projectNumber: Int!
  ...
}

The identity claim currentProjectId is set by the pre-token-generation trigger Lambda. I find this method the most elegant and use it in my projects.

The code for the lambda might look something like this:

import AppsyncClient from 'appsync-client';
import * as lambda from 'aws-lambda';
import {
  CreateUserDocument,
  GetUserDocument,
  Role,
} from './../../stueli/src/lib/api';

const { APPSYNC_URL } = process.env;

const client = new AppsyncClient({ apiUrl: APPSYNC_URL });

/**
 * https://www.npmjs.com/package/appsync-client
 * @param event
 */
export async function handler(event: lambda.PreTokenGenerationTriggerEvent) {
  console.debug(`event: ${JSON.stringify(event)}`);

  const { getUser } = await client.request({
    query: GetUserDocument,
    variables: {
      email: event.request.userAttributes.email,
    },
  });

  console.debug(`getUser=${JSON.stringify(getUser ?? {})}`);

  event.response.claimsOverrideDetails = {
    claimsToAddOrOverride: {
      ...(getUser?.currentProjectId
        ? { currentProjectId: getUser.currentProjectId }
        : {}),
    },
  };

  console.debug(
    `event.response.claimsOverrideDetails=${JSON.stringify(
      event.response.claimsOverrideDetails,
    )}`,
  );

  return event;
}

The Lambda determines first which Project the user may access with currentProjectId and then it sets the claim currentProjectId:1234. Now must be implemented of course still like the user the currentProjectId at all change and/or set can and like afterward the JWT Token is reloaded.

In my case, this happens when the user clicks on the project via the React router. First, the currentProjectId entry is made in the user table and then the JWT token is reloaded using the JWT Refresh token. If you would like to have this in detail, please write me.

Conclusion

I have presented here different methods of how permissions can be realized with the Amplify AppSync directive @auth. If you have any other cool ideas, feel free to let me know.

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on the:

And don't forget to visit my site

Building a Prototype with ninox.com for your SaaS Product

Sun, 29 Jan 2023 00:00:00 GMT

Hi,

I am very excited to work on our SaaS product. In the Minimalempires Community I found a great CoFounder with a great idea. With the low-code platform ninox.com we want to validate this idea and build a prototype. So far I had no experience with low-code and could not imagine that such would be useful for me, since I already have a lot of experience with frontend and backend. I was clearly taught better! In the next paragraphs I will explain what ninox.com is and how it helps us to develop a prototype.

ninox.com

Ninox.com is a cloud-based low-code platform for building software solutions for small and medium-sized businesses. Ninox was founded in 2013 by Frank Böhmer in Germany. As of 2021, Ninox had more than 6,000 customers. Also worth mentioning is the YouTube channel of Ninox with lots of content about Ninox. As well as the weekly Open Office Hours in English and German. Here customers can ask their questions and problems about ninox to a ninox employee*.

My experience with ninox.com

The development of a prototype is super fast with ninox.com. The definition of the table, i.e. which columns and column types the respective table has, also serves to create the UI. Thus, a data model and a UI based on it can be developed super fast. Ninox.com has come up with some really cool and smart ideas on how to connect a database with a configurable UI in a meaningful and native way.

Also the already existing user management works very well. With it, simple roles and permission schemes can be mapped well.

The community of ninox.com is very active and super helpful. They always helped me when I had problems with the implementation. I only had problems with the implementation when I had to use the ninox.com native programming language. This was the case, for example, when the database needed queries and had to do certain calculations on them.

With ninox.com to the complete product?

No! We already encounter limitations that would be easier to fix with traditional tech stacks like MERN. Also the lack of a Git integration which makes the development harder because important tools like versioning and pull requests are missing.

Conclusion

Ninox is great for rapid prototyping. Beyond that it is probably not suitable for the development of a SaaS product. That was and is not the use case of ninox.com which is why that is totally ok. Nevertheless, I am super impressed with how well it worked for our prototype and am more than willing to use ninox.com again!

Many thanks also to my CoFounder. I super enjoy working with you on our product, vision and future. You are super motivated and dedicated. I'm really looking forward to our future collaborations in 2023 :)!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on the:

And don't forget to visit my site

TypeScript Lambda with cdktf

Sat, 17 Dec 2022 00:00:00 GMT

Hi,

In this blog post, I want to briefly explain what cdktf is and how you can use it to create a TypeScript Lambda. The motivation for this came from a StackOverflow post.

What is cdktf?

The Cloud Development Kit for Terraform (cdktf) is a toolkit for building and managing cloud infrastructure like AWS or Azure with Terraform. It allows you to define the infrastructure using a programming language like TypeScript or Python.

Setup cdktf

You can find all the code in my repository here. However, I'll briefly describe how the repository was created. Initialize your cdktf repo with:

cdktf init --template="typescript"
cdktf provider add "aws@~>4.0"

Optionally you can add prettier and linter. In most of my projects, I use these because they allow me to develop quickly without having to worry about formatting.

I use the Community Terraform Lambda Module to define the lambda. It allows me to define a lambda in just a few lines that are configured with a role and can also be easily extended with policies. The cool thing is that cdktf supports a type import. To do this, simply add the following module to the cdktf.json file:

"terraformModules": [
    {
      "name": "lambda",
      "source": "terraform-aws-modules/lambda/aws",
      "version": "~> 3.0"
    }
  ],

Then the command cdktf get is executed:

cdktf get

Now the module can be used in the cdktf code:

import { lambda } from './../.gen/modules/lambda';
...

If lambda has its own dependencies, they still need to be installed with:

cd src/lambda
npm install

Deploy the cdktf stack with:

cdktf deploy

Code

The lambda can then be integrated into main.ts, for example:

import { NodejsFunction } from './constructs/nodejs-function';

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name);

    const code = new NodejsFunction(this, 'code', {
      path: path.join(__dirname, 'lambda/filter-aurora.ts'),
    });

    new Lambda(this, 'FilterAuroraEventsLambda', {
      functionName: 'filter-aurora',
      handler: 'filter-aurora.handler',
      runtime: 'nodejs14.x',
      sourcePath: code.bundledPath,
      timeout: 15 * 60,
      attachPolicyStatements: true,
      policyStatements: {
        kms: {
          effect: 'Allow',
          actions: ['*'],
          resources: ['*']
        },
        s3: {
          effect: 'Allow',
          actions: ['s3:*'],
          resources: ['*'],
        },
      },
    });
  }
}

const app = new App();
new MyStack(app, 'cdktf-lambda');
app.synth();

So I use the custom construct NodejsFunction to bundle the code from TypeScript into JavaScript and show the lambda where to find the bundled JavaScript code. The NodejsFunction Construct looks like that:

import { AssetType, TerraformAsset } from 'cdktf';
import { Construct } from 'constructs';
import { buildSync } from 'esbuild';
import * as path from 'path';

export interface NodejsFunctionProps {
  readonly path: string;
}

const bundle = (workingDirectory: string, entryPoint: string) => {
  buildSync({
    entryPoints: [entryPoint],
    platform: 'node',
    target: 'es2018',
    bundle: true,
    format: 'cjs',
    sourcemap: 'external',
    outdir: 'dist',
    absWorkingDir: workingDirectory,
  });

  return path.join(workingDirectory, 'dist');
};

export class NodejsFunction extends Construct {
  public readonly asset: TerraformAsset;
  public readonly bundledPath: string;

  constructor(scope: Construct, id: string, props: NodejsFunctionProps) {
    super(scope, id);

    const workingDirectory = path.resolve(path.dirname(props.path));
    const distPath = bundle(workingDirectory, path.basename(props.path));

    this.bundledPath = path.join(
      distPath,
      `${path.basename(props.path, '.ts')}.js`,
    );

    this.asset = new TerraformAsset(this, 'lambda-asset', {
      path: distPath,
      type: AssetType.ARCHIVE,
    });
  }
}

As can be seen, esbuild bundles the TypeScript code to JavaScript code every time cdktf deploy is executed.

Conclusion

Building TypeScript lambdas with cdktf require a bit more effort compared to aws cdk TypeScript lambdas. Still, that effort is kept in check and I've shown you how to do it here. If you found the post helpful, please let me know!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on the:

And don't forget to visit my site

Terraform CI/CD Staging Pipeline with CircleCI

Sat, 01 Oct 2022 00:00:00 GMT

Hi,

Terraform CI/CD staging pipelines, where the CD stands for Continuous Deployment, allow a secure and continuous development of infrastructure with Terraform. Typically, a commit in the main branch starts the pipeline and the desired changes are first executed on the DEV stage. Through a manual approval, the changes can then be rolled out to the QA Stage and then to the PROD Stage with the pipeline.

I would like to present here how such a Terraform CI/CD staging pipeline with AWS as cloud provider can look like.

Why a CI/CD staging pipeline?

Together with the customer we plan and develop a complex AWS setup with Hasura, RDS, ECS and many other AWS services. Among other things, this setup should also interact with the existing production environment in Salesforce. So it is of utmost importance to test new functionalities on a DEV and QA environment.

These environments DEV and QA should behave as similar as possible to the PROD environment. Furthermore, changes should be executable quickly on the PROD environment with as few manual steps as possible. Exactly for these requirements a CI/CD staging pipeline is suitable.

Multi-Account Setup

To create a CI/CD staging pipeline it is best practice to use a separate AWS account for each stage like DEV, QA or PROD. So the pipeline setup must be able to bootstrap or initialize these accounts with Terraform and then roll out changes.

For the multi-account setup I used the article Terraform AWS Multi-Account Setup as a model, where config files like accounts/dev/backend.conf and accounts/dev/terraform.tfvars are used to configure the respective stage. Super cool now is that this allows the same main.tf to be used for all staging environments.

Too often I see customers working with Terraform creating separate TF files for each staging environment and copying resources back and forth. This needs to stop and the approach described here can help!

CircleCI Staging Pipeline

CircleCI is a continuous integration and continuous delivery platform for DevOps functionalities. Similar to other DevOps platforms like Travis or GitHub Actions, a pipeline yaml definition is created in the .circleci folder named config.yml. What I find very great is that CircleCI supports manual approvals and makes them easy to integrate with:

workflows:
  version: 2
  plan_approve_apply:
    jobs:
      - dev-plan-apply
      - dev-hold-apply:
          type: approval
          requires:
            - dev-plan-apply
     ...

This way the planned infrastructure changes can first be reviewed and then implemented. You can see the complete code here.

Outlook

Next I want to introduce CDKTF so that the infrastructure can be defined with TypeScript. Similar to what was the case for me with AWS CDK, I expect this to speed up development with Terraform. The type support allows me to identify missing Terraform properties early on. Also, the documentation of the properties is very convenient and I hardly have to look in the Terraform documentation.

After that, I'm also very excited to see how AWS resources like Aurora, ECS, VPC and so on integrate with Terraform. The whole thing expands my set of tools with which I can build cool stuff in AWS.

Conclusion

A CI/CD staging pipeline is an important tool to build reliable infrastructure in AWS. Here in this article, I have shown you how to do it with Terraform in AWS. Since I am still a beginner in Terraform, I may have made small mistakes. If you have any suggestions for improvement or want to discuss simple cool projects with me, feel free to write to me :)!

Thanks to the DeepL translater (free version) for helping with translating to English and saving me tons of time :).

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

AWS CDK and Amplify runtime-config

Thu, 25 Aug 2022 00:00:00 GMT

Hi,

Seamless integration of AWS CDK and Amplify apps used to be very cumbersome! With a runtime-config for the Amplify frontend React app, it's now much easier. Here I would like to introduce you to the idea of a runtime-config.

In my fullstack projects, I regularly use AWS CDK as backend. AppSync as a GraphQL implementation is the interface between the frontend and backend. The frontend is usually a React SPA (Single Page Application) hosted in an S3 bucket. I use AWS Cognito to manage and authenticate the users and usually configure the frontend React app using AWS Amplify.

Idea runtime-config

The runtime-config allows you to configure Amplify after the build phase on runtime. The dist folder of the SPA is given a file like runtime-config.json in the public folder which fetched ruding the runtime of the app. Here you see an example for the runtime-config.json:

{
  "region": "eu-central-1",
  "identityPoolId": "eu-central-1:cda9c404-0e74-439d-b40c-90204a0e1234",
  "userPoolId": "eu-central-1_Uv0E91234",
  "userPoolWebClientId": "1t6jbsr5b7utg6c9urhj51234",
  "appSyncGraphqlEndpoint": "https://wr2cf4zklfbt3pxw26bik12345.appsync-api.eu-central-1.amazonaws.com/graphql"
}

The runtime-config is then dynamically loaded in the React app via useEffect and fetch:

useEffect(() => {
    fetch('/runtime-config.json')
      .then((response) => response.json())
      .then((runtimeContext) => {
        runtimeContext.region &&
          runtimeContext.userPoolId &&
          runtimeContext.userPoolWebClientId &&
          runtimeContext.identityPoolId &&
          Amplify.configure({
            aws_project_region: runtimeContext.region,
            aws_cognito_identity_pool_id: runtimeContext.identityPoolId,
            aws_cognito_region: runtimeContext.region,
            aws_user_pools_id: runtimeContext.userPoolId,
            aws_user_pools_web_client_id: runtimeContext.userPoolWebClientId,
            aws_appsync_graphqlEndpoint: runtimeContext.appSyncGraphqlEndpoint,
            aws_appsync_region: runtimeContext.region,
            aws_appsync_authenticationType: 'AMAZON_COGNITO_USER_POOLS',
            Auth: {
              region: runtimeContext.region,
              userPoolId: runtimeContext.userPoolId,
              userPoolWebClientId: runtimeContext.userPoolWebClientId,
              identityPoolId: runtimeContext.identityPoolId,
            },
          });
      })
      .catch((e) => console.log(e));
  }, []);

As you can see, a fetch to load the runtime-config.json is executed initially. After that Amplify is configured with the extracted properties.

You can also use HTML window variables to set the Amplify parameters. However, I prefer the fetch solution presented here because it is potentially more responsive to a missing runtime-config.json or single missing properties. Also, window variables should be avoided as they get global access to the DOM.

Workflows

The typical workflow without the runtime-config to build and deploy the React app sometimes went like this:

curl and store current endpoints like user pool id, AppSynch endpoint and more.
build Amplify config file
build react app
cdk deploy react dist folder to S3

Build pipeline workflow with runtime-config:

build react app
cdk deploy react dist folder and runtime config to S3

CDK example

The complete code is available in my GitHub Senjuns project.

const userPool = new cognito.UserPool(...)
...
const identityPool = new cognito.CfnIdentityPool(...)
...

const dashboard = new StaticWebsite(this, 'dashboard', {
    build: '../dashboard/build',
    recordName: 'dashboard',
    domainName: props.domainName,
    runtimeOptions: {
        jsonPayload: {
            region: core.Stack.of(this).region,
            identityPoolId: identityPool.ref,
            userPoolId: userPool.userPoolId,
            userPoolWebClientId: userPoolWebClient.userPoolClientId,
            appSyncGraphqlEndpoint: graphqlUrl.stringValue,
        },
    },
});

The StaticWebsite is a simple L3 CDK construct with an S3 static website bucket as the main resource. You can see more details here. But the interesting details are in the runtimeOptions object. There the endpoints for the runtime config for Amplify are stored. Behind this is the S3 Bucket Deployment Construct which transfers the endpoints via s3deploy.Source.jsonData(...) into the JSON file runtime-config.json:

const DEFAULT_RUNTIME_CONFIG_FILENAME = 'runtime-config.json';

...

new s3deploy.BucketDeployment(this, 'BucketDeployment', {
    sources: [
    s3deploy.Source.asset(props.build),
    ...(props.runtimeOptions
        ? [
        s3deploy.Source.jsonData(
            props.runtimeOptions?.jsonFileName ||
                DEFAULT_RUNTIME_CONFIG_FILENAME,
            props.runtimeOptions?.jsonPayload,
        ),
        ]
        : []),
    ],
    distribution,
    destinationBucket: siteBucket,
});

This is a cool CDK integration :) ! Just giving the BucketDeployment Construct the two parameters like the React dist and the runtime-config is a pretty smart idea.

Workaround with nested stack outputs

During my work with the runtime-config I encountered a problem. It is not possible to use CDK outputs from a nested stack for runtime-config. But there is a workaround using AWS Systems Manager parameters:

const graphqlUrl = new ssm.StringParameter(this, 'GraphqlUrl', {
    parameterName: 'GraphqlUrl',
    stringValue: appSyncTransformer.appsyncAPI.graphqlUrl,
});

...

const dashboard = new StaticWebsite(this, 'dashboard', {
    build: '../dashboard/build',
    recordName: 'dashboard',
    domainName: props.domainName,
    runtimeOptions: {
    jsonPayload: {
        region: core.Stack.of(this).region,
        identityPoolId: identityPool.ref,
        userPoolId: userPool.userPoolId,
        userPoolWebClientId: userPoolWebClient.userPoolClientId,
        appSyncGraphqlEndpoint: graphqlUrl.stringValue,
    },
    },
});

Cool, right? The nested stack output is simply stored in an SSM string parameter and can then be read later. Thanks to Adrian Dimech for the great workaround 🙏.

Conclusion

AWS CDK and Amplify are a powerful combination. With the runtime-config presented here, this combination feels much better! I copied this solution from aws-prototyping-sdk. There are some interesting AWS CDK constructs being developed in this repo. So you should definitely check it out!

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 32

Sun, 14 Aug 2022 00:00:00 GMT

In this weekly newsletter, I write a summary of posts that got my attention during the week. I center around topics AWS, DevOps, Architecture, AWS CDK, Serverless, Data Engineering, Data Analytics and much more.

If you like my content follow me on my usual spots to hang around:

Hey CDK, how do cross-account deployments work?

https://garbe.io/blog/2022/08/01/hey-cdk-how-to-cross-account-deployments/

A nice deep dive into how CDK bootstrap works and which roles for what purpose it creates. Furthermore the author explains the different options for how to deploy a CDK stack and when your should use which method. Read it!

Thanks a lot to the author Philipp Garbe

Why Developers Are Building So Many Side Projects

https://future.com/developers-side-projects/

Why developers are working or should work on side projects? It is super exciting to read why developers are working on side projects. I myself am working on side projects as well and it is a lot of fun :). Though my dream would be to get a somewhat successful product running.

Thanks a lot to the author Ben Stokes

3 Things to Know Before Building a Multi-Tenant Serverless App

https://www.readysetcloud.io/blog/allen.helton/things-to-know-before-building-a-multi-tenant-serverless-app/

This article is around the topic of multi-tenancy in serverless applications. In this, the author reports about his past experience when working on implementing a multi-tenant solution and which problems he has encountered. Super interesting read!

Thanks a lot to the author Allen Helton

SHOULD YOU PAY FOR CI/CD IN 2022?

https://blog.symops.com/2022/08/10/pay-for-continuous-integration/

Gives a historical roundtrip of the CI/CD history.

Thanks a lot to the author Adam Buggia and Brian Tarbox

Amazon CloudFront for your tech stack

https://dev.to/aws-builders/amazon-cloudfront-for-your-tech-stack-5ggk

A nice article about one of my favorite AWS services CloudFront. It explains what you can do with CloudFront like reducing latency via caching, implementing geo restrictions, and much more.

Thanks a lot to the author Jones Zachariah Noel

Title

https://dev.to/logrocket/micro-frontend-with-react-and-nextjs-n6h

Explains an interesting way how to create micro frontends with react and nextjs.

Thanks a lot to the author

How to develop strong problem-solving skills as a software developer

https://dev.to/nathan20/how-to-develop-strong-problem-solving-skills-as-a-software-developer-25nb

Offers a good summary of different ways how to solve complex problems as a software developer (probably not just there)! Further, the author explains how to improve your problem-solving skills.

Thanks a lot to the author Nathan

Thinking like containers

https://dev.to/leandronsp/thinking-like-containers-3k24

Explains very on the point what containers are about. And how to persist data from the container via volumes is described very well as well.

Thanks a lot to the author Leandro Proença

Low-cost best practices in AWS

https://dev.to/aws-builders/low-cost-best-practices-in-aws-234

A very good article that highlights best practices in terms of security and saving costs.

Thanks a lot to the author guilleojeda

I got my first Pull Request merged

https://dev.to/jhhornn/i-got-my-first-pull-request-merged-3ei9

A super encouraging article about first-time contributions to an open source project. So cool!

Thanks a lot to the author Awosise Oluwaseun

Make Errors User-Friendly with React Error Boundaries

https://dev.to/thisurathenuka/make-errors-user-friendly-with-react-error-boundaries-9e7

An interesting read of React Error Boundaries for rendering JavaScript errors in a nicer way.

Thanks a lot to the author Thisura Thenuka

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 31

Sun, 07 Aug 2022 00:00:00 GMT

This week was my first week solely working as a freelancer. That is scary but exciting too :)!

If you like my content follow me on my usual spots to hang around:

Aurora Serverless v2 PostgreSQL vs. provisioned Aurora PostgreSQL

https://towardsaws.com/aurora-serverless-v2-postgresql-vs-provisioned-aurora-postgresql-f60a93b99b83

That is an interesting article that compares the costs from Aurora Serverless v2 PostgreSQL vs Aurora PostgreSQL. And as you can read in the article it isn't always cheaper! Actually, it seems Aurora Serverless should be avoided when you have high DB traffic. So therefore perhaps use it for development and later consider switching to Aurora on-demand!

Thanks a lot to the author Ian Binder

CDK: Certificate Handling with Custom Resources

https://blog.56k.cloud/cdk-certificate-handling-with-custom-resources/

In this article, the author uses AWS CDK Custom Resources to solve a problem with DNS validating a certificate. As the custom domain isn't managed by Route 53 you can't use the standard route53 constructs to do that. Super smart how the author solved it by using a custom resource.

Thanks a lot to the author Jochen Zehnder

Containers vs Serverless

https://greengocloud.com/2018/10/07/Containers-vs-Serverless/

Super experiment article about containers vs serverless. It gives a short introduction to each and then the author gives his opinion. I agree with the author. If you want to know whom we are betting on then read the article!

Thanks a lot to the author Blake Green.

Allow only what your Lambda code needs

https://dev.to/aws-builders/allow-only-what-your-lambda-code-needs-59f2

A nice article to explain the necessity of least-privilege IAM permissions.

Thanks a lot to the author Jones Zachariah Noel

Pros and Cons of Public Cloud WAFs

https://thenewstack.io/pros-and-cons-of-public-cloud-wafs/

A critical summary of public Cloud WAF and when you should and shouldn't use them.

Thanks a lot to the author Alessandro Fael Garcia

It's never too late: Mechanical engineer to Web developer

https://dev.to/sagarpanchal0055/its-never-too-late-mechanical-engineer-to-web-developer-58di

An impressive short story from an engineer changing career from a mechanical engineer to a web developer. Very cool and inspiring :)!

Thanks a lot to the author Sagar Panchal

Styled Components in Material UI (MUI) with Styled Utility

https://dev.to/rasaf_ibrahim/styled-components-in-material-ui-mui-with-styled-utility-3l3j

A nice example of how to use React-styled components together with MUI / (former Material UI).

Thanks a lot to the author Rasaf Ibrahim

What Does The Future Hold For Serverless?

https://www.readysetcloud.io/blog/allen.helton/what-does-the-future-hold-for-serverless/

Super exciting article is about the future development of serverless. I totally agree that more analytic tools/capabilities on how to optimize the architecture would be great.

Thanks a lot to the author Allen Helton.

Is AWS support really this terrible?

https://www.reddit.com/r/aws/comments/wf79q9/is_aws_support_really_this_terrible/

An interesting discussion about how the support with AWS is and how it can be improved.

Selling an AMI and a CloudFormation template as an alternative to SaaS

https://cloudonaut.io/selling-ami-cloudformation-alternative-saas/

Interesting article on how to use the AWS Marketplace with AMI and CloudFormation type to earn money. As well the story of product marbot from cloudonaut.io is super fascinating.

Thanks a lot to the author Andreas Wittig

DynamoDB Pros & Cons

https://dev.to/roy8/dynamodb-pros-cons-k2a

A short introduction to AWS DynamoDB. Really good if you don't know much about DynamoDB.

Thanks a lot to the author Roy

Managing Secrets in Your DevOps Pipeline

https://thenewstack.io/managing-secrets-in-your-devops-pipeline/

Contains some useful guidelines when managing secrets.

Thanks a lot to the author PAVAN BELAGATTI

A Simple Guide to CI/CD For Total Beginners in 2022

https://dev.to/hackmamba/a-simple-guide-to-cicd-for-total-beginners-in-2022-2kgp

Explains the idea of CI/CD and why we need it. Gives a quick round trip for some CI/CD tools.

Thanks a lot to the author Scofield Idehen

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 30

Sun, 31 Jul 2022 00:00:00 GMT

This week was tough as it was my last week at Monada. It was a great time working with great engineers and I am super excited to follow the future development of Monada :).

If you like my content follow me on my usual spots to hang around:

Accessibility Check #1

https://dev.to/yuridevat/accessibility-check-1-3c38

The author Julia tells about her first experience with accessibility checking. It seems quite convenient to check the website using the article mentioned DevTool extension.

Storage-First Serverless Solutions :rocket:

https://leejamesgilmore.medium.com/storage-first-serverless-solutions-c0fc182243b9

A cool article about how to build a modern asynchronous API with the storage-first approach where the request is saved in the storage to optionally play it again in the case of further processing fails. The author describes a simple example and provides a lot of useful Lambda and AWS CDK code. Great article!!

Thanks a lot to the author Lee James Gilmore

Increase Your Coding Productivity with Amazon CodeWhisperer

https://medium.com/@IVaibhavMalpani/increase-your-coding-productivity-with-amazon-codewhisperer-7d4ed1b54295

If you are using Python or JavaScript for your business logic give Amazon CodeWhisperer a try. It seems super worthwhile!

Thanks a lot to the author Vaibhav Malpani

Introduction to AWS Step Functions

https://dev.to/aws-builders/introduction-to-aws-step-functions-28k7

That is a nice overview of AWS Step Functions. The author explains when to use it and what are the main features.

Thanks a lot to the author Avik Kundu

How to Automate REST API Security Testing

https://dev.to/intesar/how-to-automate-rest-api-security-testing-4lh4

Introduces some interesting tools for automated REST API security testing.

Thanks a lot to the author Intesar Shannan Mohammed

DevOps Toolchain for Beginners

https://dev.to/vishnube/devops-toolchain-for-beginners-2aif

A super nice in-depth article about useful toolchains for developers. A must-read if you are new to the DevOps space!

Thanks a lot to the author Vishnu Vasudevan

Introduction of Facade Pattern in Javascript

https://dev.to/yohanesss/introduction-of-facade-pattern-in-javascript-2gmo

A nice and tight article explaining the Facade pattern with a JavaScript example.

Thanks a lot to the author Yohanes Setiawan

How do you/company deal with secrets on ci/cd?

https://www.reddit.com/r/devops/comments/wajare/how_do_youcompany_deal_with_secrets_on_cicd/

Interesting Reddit discussion on how to deal with secrets on CI/CD. I'll favor AWS SecretsManager for staying in the tool space I already use.

Serverless Diary: How to Increase Agility with Feature Toggles — Classical Approach

https://kothiyal-anuj.medium.com/serverless-diary-how-to-increase-agility-with-feature-toggles-classical-approach-ae5f197e9be4

Interesting article on how you could implement feature toggles or flags in a serverless environment. Lists as well pros and cons when using the feature flags as part of AWS Lambda.

Thanks a lot to the author Anuj Kothiyal

Solutions Architect Tips - How to Design Applications For Growth

https://www.readysetcloud.io/blog/allen.helton/how-to-design-applications-for-growth/

A nice top-down article on how to design your application for growth.

Thanks a lot to the author Allen Helton

7 Best Tips For Web Developers

https://dev.to/devsimc/7-best-tips-for-web-developers-636

Some super cool tips for web developers. In the comment section are some more cool and useful tips :).

Thanks a lot to the author simc dev CSM® and the commenters.

What is Continuous Cloud Compliance?

https://www.mitigant.io/blog/what-is-continuous-cloud-compliance

Compliance in the cloud is super interesting. This article explains the different kinds of compliance standards. Furthermore, the author drafts the motivation behind cloud compliance.

Thanks a lot to the author Kennedy Torkura

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 29

Sun, 24 Jul 2022 00:00:00 GMT

This week was hot in Germany. Not just in terms of cool news around DevOps and more, it was physically hot :). My highlight of this week was the in-person AWS UG Meetup in Berlin. It was super amazing to see some AWS CDK friends in real :)!

If you like my content follow me on my usual spots to hang around:

:astonished: My Top 5 List of Resources that I Wish I Knew When I Started Programming

https://dev.to/eludadev/my-top-5-list-of-resources-that-i-wish-i-knew-when-i-started-programming-j04

Contains some useful tips for how to improve your programming skills. My favorite one is to connect with a developer community. One community my heart belongs to is the AWS CDK community. Come and join us. Is a lot of fun and helpful people. https://join.slack.com/t/cdk-dev/shared_invite/zt-xtpfmrqt-6ormYTA0hLdpMSAtTkM_2A

Thanks a lot to the author Eluda

Lambda Powertools TypeScript is Generally Available

https://dev.to/aws-builders/lambda-powertools-typescript-is-generally-available-1dm8

A super nice deep dive into the Lambda PowerTools TypeScript and its features. The structured log capability looks super interesting. And I didn't know about the embedded metric for CloudWatch. That is super cool!

Thanks a lot to the author Matt Morgan

What Is Data Gravity And Is It Important With Serverless?

https://www.readysetcloud.io/blog/allen.helton/what-is-data-gravity-and-is-it-important/

An interesting article on how accumulating data (data gravity) makes it hard to move like potentially changing the cloud provider. The author explains how to reduce the burden of data gravity.

Thanks a lot to the author Allen Helton

AWS open-source news and updates, #121

https://dev.to/aws/aws-open-source-news-and-updates-121-1h18

As always a very good summary of the week. The spot-interruption-simulation project got my attention. It sounds super cool for simulating the use of Ec2 spot instances.

Thanks a lot to the author Ricardo Sueiras

Simplifying serverless best practices with AWS Lambda Powertools for TypeScript

https://aws.amazon.com/blogs/compute/simplifying-serverless-best-practices-with-aws-lambda-powertools-for-typescript/

A summary of the AWS Lambda Powertools for TypeScript. This looks very interesting and for my next more complex Lambda, I will give it a try!

Thanks a lot to the author Julian Wood

The truth behind being a good programmer

https://dev.to/bekbrace/the-truth-behind-being-a-good-programmer-171b

Yep, it has fair points for what makes a good programmer :).

Thanks a lot to the author Bek Brace

What Software Developers can learn from Gutenberg

https://dev.to/polterguy/what-software-developers-can-learn-from-gutenberg-5bpa

A sweet enlightenment of how important Johannes Gutenberg was to the software development community and everything else.

Thanks a lot to the author Thomas Hansen

Why do I think snapshot tests are a bad idea

https://dev.to/syeo66/why-do-i-think-snapshot-tests-are-a-bad-idea-1imp

An interesting explanation of why snapshot testing is a bad idea.

Thanks a lot to the author Red Ochsenbein

5 Questions to Ask When Developing an Automation Strategy

https://thenewstack.io/5-questions-to-ask-when-developing-an-automation-strategy/

A nice top-down article about automation when developing. But it feels a bit like fitting into a context of a bigger software company.

Thanks a lot to the author CHRIS TOZZI

5 things to keep in mind as a programmer

https://dev.to/devsimc/5-things-to-keep-in-mind-as-a-programmer-1c24

A routine of continuous learning is my favorite out of those 5. But the others are important too so read it!

Thanks a lot to the author simc dev

Single Sign-On (SSO) using AWS Cognito and Azure AD

https://blog.getambassador.io/single-sign-on-sso-using-aws-cognito-and-azure-ad-356951536218

Interesting to see how you can leverage your Azure Active Directory (AD) for setting as Identity Provider (IdP) for AWS Cognito.

Thanks a lot to the author Supratip Banerjee

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 28

Sun, 17 Jul 2022 00:00:00 GMT

If you like my content follow me on my usual spots to hang around:

Automating AWS account bootstrapping with AWS-CDK

https://medium.com/@andrewfrazer/automating-aws-account-bootstrapping-with-aws-cdk-117e5ead1c51

Super interesting article on how to CDK bootstrap new accounts in a more DevOps way. Very cool is the idea that whenever a new account is created with Control Tower that will trigger an AWS CodeBuild job to bootstrap the account.

Thanks a lot to the author Andrew Frazer. And thank you for being a very active member of the CDK Community.

The Cost of Production Blindness

https://dev.to/codenameone/the-cost-of-production-blindness-48aj

Super nice input in one of my favorite topics production. I totally agree that we need to peek more into production. What is the problem when you don't do that and some attempts how to do it are explained in this article.

Thanks a lot to the author Shai Almog

Why Do Developers Prefer Linux?

https://dev.to/techmaniacc/why-do-developers-prefer-linux-32a3

An interesting explanation of why developers prefer Linux. If you only compare Windows and Linux, I tend to agree with the points but the comparison with macOS is missing. Sure you could argue that macOS is close to Linux.

Thanks a lot to the author Joseph Mania

Streams in NodeJS

https://dev.to/kalashin1/streams-in-nodejs-2b8l

A nice article about the concept of streams. I used them a couple of times like in Data Engineering projects as I find them super useful for effectively processing big text data. In special the piping feature for connecting streams is super cool.

Thanks a lot to the author Kinanee Samson

Two ways to directly integrate AWS Lambda function with Amazon API Gateway

https://dev.to/aws-builders/two-ways-to-directly-integrate-aws-lambda-function-with-amazon-api-gateway-3can

The article explains the two different types/ways how to connect an AWS API Gateway with a Lambda. That is the proxy and non-proxy type. Usually, you are using the proxy type. TBH. I never used the non-proxy type so it is interesting to read about it!

Thanks a lot to the author Wojciech Matuszewski

The best password manager on Linux?

https://dev.to/robole/the-best-password-manager-on-linux-f3c

Highlights KeePassXC as noncloud hosted password manager. I use it for some years now and I like it too :). I use it for Mac laptops as well.

Thanks a lot to the author Rob OLeary

Programming Myths vs Facts

https://dev.to/qavsdev/programming-myths-vs-facts-5cja

A nice Myths vs Facts for programming. It gives a better insight into what programmers are doing and clarifies some wrong expectations. I like!

Thanks a lot to the author QAvsDEV

Help - A little overwhelmed in DevOps

https://www.reddit.com/r/devops/comments/vwgu5b/help_a_little_overwhelmed_in_devops/

A nice discussion to give a kind of a small roadmap for DevOps and helps to understand wrong expectations about DevOps.

AppSync and DynamoDB Lessons Learned

https://instil.co/blog/appsync-dynamodb-lessons-learned/

Exciting discussion with like DynamoDB single vs multi-table design. I personally favor multi-table design supported with Amplify AppSync.

Thanks a lot to the author Matthew Wilson

History of APIs

https://dev.to/littleironical/history-of-apis-281j

An interesting history roundtrip about APIs

Thanks a lot to the author Hardik kumar

Investigating 15s HTTP response time in AWS ECS

https://dev.to/gwenshap/investigating-15s-http-response-time-in-aws-ecs-2gge

An interesting story about investigating a long response time when using AWS ECS in multiple availability zones.

Thanks a lot to the author Gwen (Chen) Shapira

How to ask for help

https://dev.to/dhravya/how-to-ask-for-help-2690

Super article about how to ask correctly for help in like a community.

Thanks a lot to the author Dhravya

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Catch AWS Lambda Timeouts

Sun, 10 Jul 2022 00:00:00 GMT

(Click me :D)

Ahoy,

AWS Lambda is probably one of the most famous if not the most famous compute service from AWS. In fact, Lambda is also my favorite AWS service because it makes implementing business logic very easy and straightforward. However, a few times I had the problem that the timeout of Lambda was too low and thus the code was not completely executed. Debugging or catching this error was also often exhausting and annoying. A good solution for me was to use the Lambda Duration Metrics to define an alarm.

import * as cw from 'aws-cdk-lib/aws-cloudwatch';
import * as lambdajs from 'aws-cdk-lib/aws-lambda-nodejs';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';

const exampleLambda = new lambdajs.NodejsFunction(this, 'ExampleLambda', {
    timeout: core.Duration.seconds(10),
});

new timeoutAlarm - new cw.Alarm(this, 'TimeoutAlarm') {
    metric: exampleLambda.metricDuration({ statistic: 'max' }),
    threshold: exampleLambda.timeout?.toMilliseconds() ?? 1,
    evaluationPeriods: 1,
}

const topic = new sns.Topic(
    this,
    'ErrorTopic',
);
topic.addSubscription(
    new subscriptions.EmailSubscription('alice@bob.com')
);
timeoutAlarm.addAlarmAction(
    new actions.SnsAction(topic)
);

As you can see here, a simple Lambda JS Function is created at the beginning. By the way the NodejsFunction construct also supports TypeScript. Then the timeout alarm is defined which uses the duration metric and the timeout value from the Lambda to configure.

After that, an Error Topic is created via which the triggered alarm is forwarded by email using an EmailSubscription.

Advantages

The advantage of this solution is that the Lambda code does not need to be modified and the problem is solved on infrastructure level or AWS CDK level. This solution can be applied to other Lambda languages like Python or Java as well.

Conclusion

Lambda timeout caching is annoying. With a little AWS CDK code and the Lambda Duration metric, the problem is quickly solved.

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 27

Sun, 10 Jul 2022 00:00:00 GMT

This week was crazy exciting. On Thursday I took the train to Munich for attending to an AWS Meetup and visit a client and friend. Both events were super empowering and in times like this, I feel very grateful and thankful to work in this area.

If you like my content follow me on my usual spots to hang around:

What Made GoLang So Popular? The Language’s Creators Look Back

https://thenewstack.io/what-made-golang-so-popular-the-languages-creators-look-back/

An interesting history telling about GoLang and how it became so popular. TBH, I didn't work much with GoLang so far so it is hard for me to see the gold. If you have a project with GoLang and need some help, let me know :)!

Thanks a lot to the author DAVID CASSEL

How to Use Feature Flags

https://dev.to/bentorvo/how-to-use-feature-flags-39md

Why and how you can use feature flags. A very appealing part for feature flags, I find the possibility to roll the feature out to production but disable it quickly if it is causing problems. As well in combination with A/B Testing feature flags can be used to roll out the changes only to a part of the customers.

Thanks a lot to the author Ben Brazier

GitHub Copilot and Open Source: A Love Story That Won’t End Well?

https://thenewstack.io/github-copilot-and-open-source-a-love-story-that-wont-end-well/

There are currently a lot of hot topics around GitHub Copilot. This article summarizes the problem very well. TBH I for myself don't have a strong position. On one hand, I like GitHub as a Git provider with super nice extra features. On the other hand, violating the taking code from a developer who didn't agree to that is awful. The conclusion from the author is nice and makes sense to me.

Thanks a lot to the author SASHA MEDVEDOVSKY

Implementing CUDOS for an Enterprise - Part 2: Filtering Data using Row Level Security

https://dev.to/aws-builders/implementing-cudos-for-an-enterprise-part-2-filtering-data-using-row-level-security-1g2p

A comprehensive article about AWS QuickSight and how to manage your QuickSight user in groups to allow them access to the QuickSight resources. It feels like good QuickSight articles or information are rarely so it is super valuable to have articles like this one.

Thanks a lot to the author Mike Graff

Building an open-source company in public

https://dev.to/novu/building-in-public-4i73

Uh, that is super attractive. Building a complete open-source company. I super like that idea. Being an open-source would include sharing the salary

Thanks a lot to the author Tomer Barnea

Cloud and the shared responsibility model misconceptions

https://dev.to/aws-builders/cloud-and-the-shared-responsibility-model-misconceptions-2lem

The author explains some misconceptions which can occur when a user starts to use a cloud provider.

Thanks a lot to the author Eyal Estrin

Open-Source contribution, a beginner's guide (in 2 minutes)

https://dev.to/mohsenkamrani/open-source-contribution-a-beginners-guide-in-2-minutes-2g58

A nice short intro to GitHub. I personally love working with GitHub.

Thanks a lot to the author mohsen

RDS Aurora Cost Comparison

https://www.reddit.com/r/aws/comments/vuxhdb/rds_aurora_cost_comparison/

A super interesting discussion about costs with AWS RDS Aurora. The general approach seems to be more bottom-up-ish with just trying it out and experiencing the costs.

Learn to code with these 5 proven learning strategies

https://dev.to/educative/learn-to-code-with-these-5-proven-learning-strategies-58gj

Some very good tips for how to improve coding and learning in general. Must read!

Thanks a lot to the author Hunter Johnson

Open Source Projects

In this section, I list Open Source projects I find super interesting!

aws-prototyping-sdk

https://github.com/aws/aws-prototyping-sdk

An amazing new-ish project from the AWS folks with some very sophisticated AWS CDK and Projen constructs. In special the stack-website construct is super mind-blowing and I could find a lot of gold to grab for my own projects. I recommend keeping that on your radar! Thank you so much!

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Make your AWS Prod environment super with Superwerker

Tue, 05 Jul 2022 00:00:00 GMT

Hi everyone.

I want to make my senjuns project production-ready. So far I had only deployed it in my developer AWS account. My developer account has no AWS services configured that are needed for a reasonable production environment. These are for example AWS SSO, ControlTower, SecurityHub and some more. The Superwerker CFN Template came in handy. It allowed me to configure a new AWS account with many great AWS services for production use. In the following sections, I would like to go into what Superwerker is exactly and how I used it.

What is Superwerker?

Superwerker is an open-source AWS CloudFormation solution that simplifies the setup of an AWS account. It follows best practices for security and efficiency. It was/is developed and maintained by AWS Advanced Partners kreuzwerker and superluminar. It even has its own landing page with great content like a short video and more. Hard to believe this is a free product.

The Superwerker deployment has a lot of cool features which are all explained very well in the docs here. I really like that it uses Architectural Decision Records to describe each feature. Below I explain the features in my own words.

Backup

Backup enables the AWS Backup Service and by default, a backup is created every day. It creates backups of databases that are also supported by AWS Backup.

Budget

Budget helps to keep track of the costs in AWS and gives a warning in time if there are more costs than usual. It does this by calculating the average of the last three months and if this is exceeded, a warning comes. A really cool idea in my opinion.

Control Tower

Control Tower simplifies managing multiple AWS accounts with AWS SSO. For me, this is the main reason why I use Superwerker. So I can access my different AWS accounts like build, dev and prod very easily. I also like the fact that with AWS SSO I can simply copy the AWS account credentials and use them for the AWS CLI.

GuardDuty

GuardDuty is an AWS service for finding potential security threats.

Living Documentation

Living Documentation is a pretty cool feature that generates documentation via the AWS CloudWatch Dashboard. This then provides information on, for example, how to do an SSO setup with external identity providers like Google and much more. Extremely cool is how this documentation also updates itself when certain configurations have already been made. That's why it's called Living Documentation.

Notifications

Superwerker creates OpsItems in OpsCenter when emails are sent from the RootMail feature. With the Notifications feature emails are sent to the specified addresses. This way the user does not have to manually log into the AWS Console and check OpsCenter. Through email, one thus also has a centralized collection point for multiple accounts.

RootMail

RootMail creates the uniformed email addresses for the audit and log archive accounts created by Control Tower. For this purpose, a separate Hosted Zone is created, for example aws.mycompany.test with Route53, and the required settings are made with AWS SES to receive the emails.

Security Hub

The Security Hub feature enables AWS Security Hub for all accounts. The user needs to log into the audited account to view the aggregated Security Hub view.

In the next section, I would like to share more about my Experience with Superwerker.

Superwerker Experience

You can find the instructions for running Superwerker here. For me, the installation went quite smoothly. However, the RootMail nested stack was an exception. I had previously imported my senjuns.com domain and associated Hosted Zone from another AWS account. However, I did not properly configure the name servers from the domain in the designated Hosted Zone. By debugging the Superwerker Lambdas, I noticed this error.

So if you also get the following error logs in the RootMailReady Lambda, you just need to check if you have configured your domain and Hosted Zone properly.

{
    "level": "info",
    "msg": "verification not yet successful",
    "res": {
        "VerificationAttributes": {
            "aws.senjuns.com": {
                "VerificationStatus": "Pending",
    ...

{
    "level": "info",
    "msg": "DKIM verification not yet successful",
    "res": {
        "DkimAttributes": {
            "aws.senjuns.com": {
                "DkimEnabled": true,
                "DkimVerificationStatus": "Pending",
    ...

If anyone encounters the same or similar problem, I have created a pull request with troubleshooting information.

Conclusion

Superwerker is a great CloudFormation QuickStart to set up an AWS production environment professionally and easily. Many improvements and new features are already planned in the Superwerker Backlog. The Superwerker community is still looking for active support in implementing new features and improvements. I am very excited to continue using the Superwerker deployment in my current project.

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

Newsletter martinmueller.dev 2022 week 26

Sun, 03 Jul 2022 00:00:00 GMT

Hello friends.

Another week passed by! My highlights of the week were the German CDK Happy Hour Meetup on Wednesday where an AWS CDK project was presented which leverages AWS ECS with Blue/Green deployments. That is super exciting as that is not natively supported by ECS CloudFormation at all. Another highlight was that I helped a friend move house and carrying furniture around like that is a good workout :).

If you like my content follow me on my usual spots to hang around:

Nobody talks about this in the IT world

https://dev.to/antoomartini/nobody-talks-about-this-in-the-it-world-31ph

An article about imposter syndrome. It is so important to speak about it more in our area. I personally never felt affected by it but I would love to help persons who do. In the discussion the community gives nice tips how to overcome the imposter syndrome.

Thanks a lot to the author Maria Antonella :butterfly:

Why developers must start blogging and how to start?

https://dev.to/atapas/why-developers-must-start-blogging-and-how-to-start-543c

Obviously, I totally agree with that developers must start blogging. The author adds good explanation why writing blogs about your work as a developer is useful. For me it was super hard to start with but it started to getting easier and now I love it! I would like to add that it helps to increase your language skills.

Thanks a lot to the author Tapas Adhikary

Setting up a CI/CD Pipeline for AWS Fargate using Github Actions - NodeJS version (Including creation of Infrastructure using AWS CDK)

https://www.cloudtechsimplified.com/ci-cd-pipeline-aws-fargate-github-actions-nodejs/

Nice in deep article about how to setup AWS ECS with Fargate with AWS CDK. That topic can be qutie difficult so it is nice to have articles like this one to help you with.

Thanks a lot to the author Mugilan Ragupathi

Large object storage strategies for Amazon DynamoDB

https://aws.amazon.com/blogs/database/large-object-storage-strategies-for-amazon-dynamodb

Article about storing large object as reference in AWS DynamoDB.

Thanks a lot to the author Josh Hart

What Is Serverless Enablement And Why Is It Important?

https://www.readysetcloud.io/blog/allen.helton/what-is-serverless-enablement/

Super interesting article about serverless. The author is explaining why serverless is useful and how you can start with it.

Thanks a lot to the author Allen Helton

The Pros and Cons of AWS Lambda Function URLs

https://dev.to/bentorvo/the-pros-and-cons-of-aws-lambda-function-urls-524d

Interesting pros and cons about the new-ish AWS Lambda Function Url.

Thanks a lot to the author Ben Brazier

Introducing the new AWS Step Functions Workflows Collection

https://aws.amazon.com/blogs/compute/introducing-the-new-aws-step-functions-workflows-collection/

Oh boy, how cool is that? Workflow collections to show best practices examples for AWS Step Functions. And you even can submit your workflows. Fantastic!

Thanks a lot to the author Benjamin Smith

The AWS CDK, Or Why I Stopped Being a CDK Skeptic

https://dev.to/mradzikowski/the-aws-cdk-or-why-i-stopped-being-a-cdk-skeptic-4o2o

A nice summary about AWS CDK and a cool story for how and why from switching from YAML to a programming language for defining infrastructure. But as well it has some valid AWS CDK cons

Thanks a lot to the author Maciej Radzikowski

Better dev experience starts from a proper document

https://dev.to/jennieji/better-dev-experience-starts-from-a-proper-document-30o5

Provide good tips on how to create proper documentation for your code projects.

Thanks a lot to the author Jennie

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 25

Sun, 26 Jun 2022 00:00:00 GMT

Hi folks.

What a week. Close by the Azure Meetup started happening again after like 2 years. Was great to see my old Meetup friends again. You might think wait what why Azure you are an AWS guy? Yes, true but learning more about the other cloud providers is super exciting and as well my new company Monada aims for multi-cloud :).

If you like my content follow me on my usual spots to hang around:

Is Zero Trust Architecture Possible??

https://www.linkedin.com/pulse/zero-trust-architecture-possible-shashank-abhishek/

An article about the zero trust architecture. It is an interesting approach. The author lists and explains the zero-trust principles. In the serverless world, some of those are implemented easily or even already part of the service. But yeah it would be nice if the developer wouldn't need to regard zero trust principles to much and they would be already part of the IaC framework. That is as well what we plan to work on in Monada :).

Thanks a lot to the author Shashank Abhishek

How Do You Overcome Imposter Syndrome?

https://dev.to/janicera2880/how-do-you-overcome-imposter-syndrome-55m8

Such an important topic as the author speaks about the imposter syndrome. It is super common in the developer's area. The author provides super nice and useful tips for how to overcome imposter syndrome. I would like to add that if you give your best every day as a developer you don't have to blame yourself! Giving your best also includes things like relaxing, detaching, time with your family, playing a game, and time for workouts. Being a developer is about you! Embrace that as I do :)!

Thanks a lot to the author Janice Alecha

Building Serverless Applications That Scale The Perfect Amount

https://dev.to/aws-builders/building-serverless-applications-that-scale-the-perfect-amount-410p

A very in-depth article about designing an architecture that scales small, medium or large. Nice idea to compare the different scales. For me, it is super exciting that with like AWS CDK it allows you with relatively low costs to change your design from small to medium or even too large. There are only a few articles like that out there so read them!

Thanks a lot to the author Allen Helton. And check out his other amazing posts which I regularly cover in my newsletters ^^.

Why you should learn TypeScript today

https://dev.to/omerwow/why-you-should-learn-typescript-today-dmo

Yeah, a super good article about TypeScript and why you should learn it. I personally love TypeScript. What I like most is that it is super spread in different areas like language for IaC for AWS CDK or AWS SDK TypeScript or React TS. Furthermore, you can use the huge ecosystem around JavaScript. As well the discussions are worth a read!

Thanks a lot to the author Omer Elbaz

What's your biggest problem while working with AWS

https://www.reddit.com/r/aws/comments/vhgmr4/whats_your_biggest_problem_while_working_with_aws/

Wow interesting discussion about the biggest problems in AWS. I find the struggle with AWS permissions interesting and provided an answer to try AWS CDK as it made juggling with permissions easier than pure with pure CFN before.

Control Cost And Security Weakness On Your AWS SandBox Accounts

https://blog.shellkode.com/control-cost-and-security-weakness-on-your-aws-sandbox-accounts-609670b1146d

Cool, the author presents a bunch of Lambda's to help clean sandbox accounts. And what is cool is the lambda is written in Python.

Thanks a lot to the author Kalaiarasu M

Open Source Projects

In this section, I list Open Source projects I find super interesting!

CDK Bill Bot

https://github.com/cremich/cdk-bill-bot

Super interesting AWS CDK collection of L2 and L3 constructs for AWS Cost and Usage reports. Definitely will try it out for the private project I am currently working on. The Creator Christian Bonzelet is a super experienced and active member of the CDK Community and an AWS Community Builder. So worth checking out!. Thanks to my friend Christian :)!

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 23

Sun, 12 Jun 2022 00:00:00 GMT

In this weekly newsletter, I write a summary of posts that got my attention during the week. I center around topics AWS, DevOps, Architecture, AWS CDK, Data Engineering, Data Analytics and much more.

If you like my content follow me on my usual spots to hang around:

As a developer, how to estimate the time for a task?

https://dev.to/ms_yogii/as-a-developer-how-to-estimate-the-time-for-a-task-44b

Article about how to estimate tasks. It explains very well why it is hard to estimate the time you will spend on a task and how you can improve that. My highlight is the advice to divide your task into smaller chunks that are less than 30 minutes in time efforts.

Thanks a lot to the author Yogini Bende

AWS Step Functions — Are Active Executions Affected When Your State Machine Gets Updated?

https://dev.to/pubudusj/aws-step-functions-are-active-executions-affected-when-your-state-machine-gets-updated-5dfn

The author highlights an interesting detail when using Lambda as part of an AWS Step Function workflow. Use Lambda Versions if you want to ensure to not update current Step Functions executions. But if you always want to update then don't use Lambda Versions. Please read the article for more details!

Thanks a lot to the author Pubudu Jayawardana

7 Pillars of Cloud Cost Optimization

https://www.linkedin.com/pulse/7-pillars-cloud-cost-optimization-shashank-abhishek/

Nice deep dive into how to optimize AWS costs. Though point 4 "Use Savings Plans" is not always a good choice in my opinion. One reason against it is that as AWS moves fast with like-new Ec2 tiers it might be better to go with on-demand and change dynamically the tier. It always depends I guess ^^. Anyway, it is a super good article :).

Thanks a lot to the author Shashank Abhishek

Seamless Integration with Remote Docker Hosts for Development

https://medium.com/geekculture/seamless-integration-with-remote-docker-hosts-for-development-7a369d94812f

Interesting article on how to run docker on a remote docker host. Though I miss a little bit of the use case or why a developer would like to do that.

Thanks a lot to the author Domenico Sibilio

Briten testen 4-Tage-Woche – von der Frittenbude bis zur Software-Firma

translated: Brits test 4-day week - from the fish and chips shop to software company

https://t3n.de/news/briten-testen-4-tage-woche-grossbritannien-1477699

Super interesting article about some companies from the UK wanting to try the 4 days week. I am a huge fan of the 4 days week. I am convinced it increases productivity in many areas.

Thanks a lot to the author Andreas Weck

Cost Optimizations in AWS - Part 1

https://dev.to/alsaheem/cost-optimizations-in-aws-part-1-25mn

Nice article about how to optimize your AWS costs.

Thanks a lot to the author Adebisi ayomide

How to create a software test plan?

https://aqua-cloud.io/create-software-testing-plan/

Super nice deep dive into testing with highlighting important requirements. As well gives you a developing test plan with concrete steps. Super useful!

Thanks a lot to the authors Denis Matusovskiy and Robert Weingartz

How did I create EC2 by Voice?

https://dev.to/aws-builders/how-did-i-created-ec2-by-voice-2p54

Super creative idea how can with the use of AWS Transcribe and Lambda create an Ec2 instance by voice. Very cool!

Thanks a lot to the author Mohamed Latfalla

Terraform vs Pulumi vs CloudFormation: Best Choice for 2022

https://docs.multy.dev/blog/iac-comparison

Very nice comparison between the Infrastructure as Code Tools like Terraform, Pulumi and CloudFormation.

Thanks a lot to the author João Coelho

Auth Portal powered by AWS/AzureAD and built with CDKs

https://faun.pub/aws-aad-auth-portal-cdk-56c782e3a048

Uh nice using AWS CDK or CDKTF to connect AWS Cognito with Azure AD. That are exciting combinations. The author as well highlights the easy testability of AWS CDK and CDKTF. Must read!

Thanks a lot to the author SEB

AWS Inspector: A Guide to Discover Your Security Holes

https://adamtheautomator.com/aws-inspector/

How to use AWS Inspector for security. Explains first how to enable the AWS Inspector. Then the author explains how to inspect Ec2 instances with it and how to interpret the findings.

Thanks a lot to the author

A Guide to AWS Certifications

https://dev.to/aws-builders/a-guide-to-aws-certifications-40ff

A deep dive into AWS certificates. Very useful for reading if you planning to do an associate or professional exam. In the end it

Thanks a lot to the author Karl Mathias Moberg

How to Improve Developer Experience: 7 Things to Change

https://dev.to/nimbusenv/how-to-improve-developer-experience-7-things-to-change-3c5e

Very nice and deep article on how to improve the developer experience. There are a lot of good tips and is hard for me to pick my favorite one. So I suggest you read them all :)!

Thanks a lot to the author Nimbus

Hugo vs. Gatsby.js

https://dev.to/cloudcannon/hugo-vs-gatsbyjs-35gd

Thanks a lot! This post is super informative. The author compares Gatsbyjs with Hugo like when you should take what. Yeah, many years ago I was choosing GatsBy because I eventually wanted to use components :). And I am about to implement components.

Thanks a lot to the author David Large

New Observability Features for AWS Step Functions

https://www.infoq.com/news/2022/06/aws-step-functions-observability

That is an interesting update about AWS Step Functions observability updates. I like the new different views Graph view, Table view and Event view.

Thanks a lot to the author Steef-Jan Wiggers

10 Secure Coding Best Practices to Follow in Every Project

https://dev.to/smartscanner/10-secure-coding-best-practices-to-follow-in-every-project-1i9h

Useful best practices for coding. My highlight is the "Process an Open-Source Spirit" where the author explains how to prepare the project to be consumable as an Open-Source project.

Thanks a lot to the author SmartScanner

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 22

Sun, 05 Jun 2022 00:00:00 GMT

In this weekly newsletter, I write a summary of posts that got my attention during the week. I center around topics AWS, DevOps, Architecture, AWS CDK, Data Engineering, Data Analytics and much more.

If you like my content follow me on my usual spots to hang around:

Cloud Computing and AWS

https://dev.to/aws-builders/cloud-computing-and-aws-14g0

Nice post to explain what cloud computing is to an audience that is not familiar with the cloud at all. It explains very nicely the cloud computing model where you divide the cloud offerings usually in Infrastructure as Code (IAAS), Platform as a Service (PAAS) and Software as a Service (SAAS). For more context please read the amazing post :)

Thanks a lot to the author Akande Olalekan Toheeb

Exploring Aurora Serverless V2 for MySQL

https://mydbops.wordpress.com/2022/05/22/exploring-aurora-serverless-v2-for-mysql/

A nice deep dive into AWS Aurora Serverless v2. Such posts are rarely as Aurora offers a wide set of cool features and explaining those is time-consuming.

Thanks a lot to the author Kabilesh P R

How do I handle brutal feedback from colleagues?

https://dev.to/sloan/how-do-i-handle-brutal-feedback-from-colleague-3oda

Discussion about bullies at the developer working place. That is of course not acceptable and the commenter is giving very good advice on how to improve the situation if you are the victim.

Are You Really API-First Or Do You Just Think It?

https://www.readysetcloud.io/blog/allen.helton/are-you-really-api-first/

Super cool post about how you design your product with API first. I totally agree with the author. I personally like to use GraphQL APIs where I just define my API as a typed schema. That has nice advantages against like the more old-fashioned REST API :).

Thanks a lot to the author Allen Helton

7 Reasons Why Serverless Encourages Useful Engineering Practices

https://www.serverless.com//blog/7-reasons-why-serverless-encourages-useful-engineering-practices

A great summary of the reasons why serverless is useful. The author proves to have much experience with serverless. The reasons are well picked and well explained. My favorite reason is number 6 "It enforces Infrastructure as Code". I encourage you to read the full article!

Thanks a lot to the author Anna Geller

Frauen in der Innovation: „Das ist keine Nettigkeit, sondern wirtschaftlicher Faktor!“

Translation: Women in innovation: "It's not a nicety, it's an economic factor!"

https://t3n.de/news/frauen-innovation-diversity-womenpower-1477339

The super interesting article makes us aware that still way too many women are part of innovation processes. It explains as well why that is bad and how to solve it.

Thanks a lot to the author Josefine Kramer

How to Easily Deploy a Spring Boot Application on AWS Fargate

https://dev.to/aws-builders/how-to-easily-deploy-a-spring-boot-application-on-aws-fargate-38nh

A small guide for how to deploy a spring boot application on AWS with AWS Fargate. Nice guide. Probably worth enhancing the deployment with AWS CDK as Infrastructure as Code (IaC) instead of clicking in the AWS Console.

Thanks a lot to the author Wilson K. KOMLAN

https://dev.to/hugomartinezzz/5-trending-vs-code-extensions-for-productivity-41di

Oh boy, there are a bunch of VS Code extension posts out there but this one blew my mind. There is a couple of extensions I need to try ASAP. Thank you so much.

Thanks a lot to the author Hugo Martinez

ALL YOU NEED TO KNOW ABOUT AMAZON ROUTE 53

https://dev.to/joshbek/all-you-need-to-know-about-amazon-route-53-8a6

Nice roundup about AWS Route 53

Thanks a lot to the author Joshua O. Berkoh

Writing and Testing a stdin script with TypeScript

https://dev.to/viglioni/writing-and-testing-a-stdin-script-with-typescript-57mb

About how to test a TypeScript Script that has stdin input. I would have needed the solution a while back as I couldn't figure out how to do that. So this article seems pretty useful.

Thanks a lot to the author Laura Viglioni

The Challenges of Building Multi Cloud

https://docs.multy.dev/blog/challenges-of-building-multi-cloud

High-level overview about building in multi-cloud and how multy helps to make it easier.

Thanks a lot to the author João Coelho

Enable Autocomplete for boto3 in VSCode

https://dev.to/aws-builders/enable-autocomplete-for-boto3-in-vscode-3jb4

How you can configure autocomplete for Python boto3 in VSCode. Super important when you develop your AWS SDK script or lambda with python!

Thanks a lot to the author Maurice Borgmeier

Is Serverless Just a Stopover for Event-Driven Architecture?

https://thenewstack.io/is-serverless-just-a-stopover-for-event-driven-architecture/

This article is super mind-blowing. I think the author is correct and serverless is kind of super related to event-driven architecture. Must read!

Thanks a lot to the author MARK HINKLE

Tips for Junior Developers

https://dev.to/sahan/tips-for-junior-developers-5ggj

Super empowering post for junior developers on how to grow as such :).

Thanks a lot to the author Sahan

A Beginner's Guide to Create AWS CDK Construct Library with projen

https://dev.to/aws-builders/a-beginner-s-guide-to-create-aws-cdk-construct-library-with-projen-5eh4

Super nice detailed post about a Projen setup for the CDK Construct library Projen type. Nice :)!

Thanks a lot to the author hayao-k

AWS Step Functions launches an interactive workshop for building and deploying application workflows

https://aws.amazon.com/about-aws/whats-new/2022/06/aws-step-functions-interactive-workshop-building-deploying-application-workflows/

An interesting AWS StepFunction Workshop. It uses AWS CDK for Infrastructure as Code (IaC).

AWS Cognito — Web Application Integration Basics

https://jawadhasan80.medium.com/aws-cognito-web-application-integration-basics-dc18b42f47c4

A cool bottom-up post about Cognito and how to use it combined with Amplify.

Thanks a lot to the author Jawadhasan

What are your debugging tips?

https://dev.to/ben/what-are-your-debugging-tips-3lgm

A super high-level and super useful discussion on how to debug efficiently. Must read!

Thanks to the author asking the question and all the commenter

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 21

Sun, 29 May 2022 00:00:00 GMT

In this weekly newsletter, I write a summary of posts that got my attention during the week. I center around topics AWS, DevOps, Architecture, AWS CDK, Data Engineering, Data Analytics and much more.

If you like my content follow me on my usual spots to hang around:

https://blog.serverlessq.com/aws-sns-vs-sqs-what-are-the-main-differences

A super good explanation about SNS and SQS and what are the differences. I never paid too much notice of what differentiates those services so I am very glad that the author is doing that. As well the author explains when to use SNS, SQS and even Kinesis. Must read!

Thanks a lot to the author Sandro Volpicella

New AWS Config Rules - LambdaLess and rusty

https://dev.to/aws-builders/new-aws-config-rules-lambdaless-and-rusty-hbe

Exciting article from my German fellow Gernot Glawe. He describes how to add more easily add custom AWS Config rules with Lambda. I didn't work much with AWS Config but this article made me excited to do so :).

Thanks a lot to the author Gernot Glawe

DevOps: An Introduction

https://dev.to/abhishe89636035/devops-an-introduction-5cdp

Gives a quick introduction to DevOps and then lists different areas in the DevOps space and tools you can use for those.

Thanks a lot to the author Abhishek Shrivastava

How to use EventBridge as a Cross-Account Event Backbone

https://dev.to/eoinsha/how-to-use-eventbridge-as-a-cross-account-event-backbone-5fik

Long post but super interesting how you can leverage AWS EventBridge for cross-account events. The author even gives a little example consisting of 3 AWS Accounts that are interchanging events via EventBridge.

Thanks a lot to the author Eoin Shanaghy

Overview of AWS Developer Tools - Part 2

https://dev.to/makendrang/overview-of-aws-developer-tools-part-2-kei

And here is the second part of the AWS Developer Tools series. My personal highlight is the AWS Fault Injection Simulator which you can use for chaos engineering testing. Super cool!

Thanks a lot to the author MakendranG

Zero Maintenance AWS Canary Tokens That Scale

https://medium.com/@williambengtson/zero-maintenance-aws-canary-tokens-that-scale-b470c6f60da

An exciting article about Honey tokens like how you can do them in AWS and how they help you to monitor malicious behavior. Very cool!

Thanks a lot to the author William Bengtson

The tale of migration to Serverless

https://dev.to/ymwjbxxq/the-tale-of-migration-to-serverless-24b3

A nice and creative sports analogy/story/tale for serverless covering how we got to serverless and why to use it.

Thanks a lot to the author Daniele Frasca

AWS open-source news and updates, #114

https://dev.to/aws/aws-open-source-news-and-updates-114-51ng

Nice summary of new AWS open-source-related projects. My highlight is auto-close-aws-accounts which as it makes deleting AWS accounts super easy!

Thanks a lot to the author Ricardo Sueiras

Please, don't give up on programming

https://dev.to/alesbe/please-dont-give-up-on-programming-4pop

Engages you to keep programming even if it seems to be tough. Totally agree with that. Try to keep your goal in front of your eyes and perhaps do a little bit every day. For more tips read the article :).

Thanks a lot to the author alesbe

AWS S3 Replication: Same Region Replication(SSR)

https://programmersblock.hashnode.dev/aws-s3-same-region-replication

A nice roundup article about S3 Same Region Replication.

Thanks a lot to the author Himanshu Gupta

Do we really need AWS?

https://dev.to/jordan_t/do-we-really-need-aws-3hao

I will not spoiler but the answer will not surprise you :P. But yeah it is a nice summary about AWS.

Thanks a lot to the author Jordan Tingling

5 VSCode Extension that makes Maintenance Easy:seedling:

https://dev.to/alexomeyer/5-vscode-extension-that-make-maintenance-easy-4eb1

Has some cool VS Code extension tips. Check it out!

Thanks a lot to the author Alex Omeyer

Kafka on the Serverless Shore: Building event-driven applications with Kafka

https://dev.to/aws-builders/kafka-on-the-serverless-shore-building-event-driven-applications-with-kafka-23df

Super interesting article about Serverless Kafka combined with AWS CDK.

Thanks a lot to the author Ibrahim Cesar

Softwareentwicklung: Dem IT-Fachkräftemangel mit interdisziplinären Teams begegnen

https://t3n.de/news/softwareentwicklung-teams-fachkraeftemangel-1475059

Interesting article in German on why interdisciplinary teams are so important and successful.

Thanks a lot to the author Elisabeth Urban

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

What is Serverless?

Tue, 24 May 2022 00:00:00 GMT

To clear up a possible misunderstanding right away! Serverless does not mean that there are no more servers! On the contrary! Cloud providers such as AWS, which offer serverless, have many servers worldwide. In the image below you can see the currently available AWS regions. This means in all these places in the world AWS has at least two data centers.

Such a data center itself is a huge building with a lot of server technology.

Now and in the following, I would like to explain for each section using S3 and Lambda as an example.

S3 is an AWS service that can be used as object storage.

Lambda is also an AWS service that can be used for compute. This is usually used to implement the business logic of the cloud application.

Why Serverless?

**Less complexity. As a cloud developer, I no longer have to worry about managing (hardware and software) the servers. So I can focus more directly on the business logic of my application in the cloud. With S3, as a developer, I can easily store data using the S3 API. With Lambda, I can define the business logic directly as code, e.g. TypeScript or Python, and pass it to the Lambda API. The AWS Lambda service then takes care of the rest. I don't need to worry about any OS related stuff like with Ec2 for example.

Better scalability. Serverless offerings scale automatically by default. So any number of objects can be stored in S3 without the developer having to adjust the volume as with AWS EBS, for example. Lambda can also scale almost arbitrarily. If the demand to execute the business logic increases, even multiple Lambda instances are started to ensure timely processing. If no execution of the respective business logic is needed, no Lambda instances are started.

Cost optimization. Already by the previous point over the better scalability it can be assumed that thereby cost savings can result to possible alternatives. But this is not always the case and it is difficult to compare costs of serverless with alternatives. Therefore, I will not discuss cost optimization in this article. I will only say that I personally like to use the serverless offerings from AWS, as this leads to very low costs, especially in the initial phase.

But how does serverless work?

If you're convinced by the reasons why Serverless works, you might be wondering how it works?

Like all AWS service offerings, communication with Serverless services like S3 and Lambda works via HTTP/S API. This means that HTTP requests can be made for creation and modification. The S3 API can be found in the documentation here and for the Lambda API here.

Now surely the developer doesn't have to bother with HTTP requests and can alternatively use quasi API wrappers like the AWS CLI or AWS SDK in different programming languages like TypeScript or Python. An even better and by me preferred alternative is AWS CDK which m

What do I like about Serverless?

The great thing about Serverless for me is that it's a great abstraction and makes developing in the cloud super easy. So I am specifically able to implement my customers' wishes and requirements for their cloud project.

Summary

In this article, I explained to you what Serverless is and why I think it's so great. However, I firmly believe that serverless is not the end of cloud abstractions. I recently joined a company called Monada that is aiming to do just that. We want to create an even better abstraction than Serverless to make working with the cloud even more awesome. If you have any feedback on this article please let me know :)! Let's build!

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

Newsletter martinmueller.dev 2022 week 20

Sun, 22 May 2022 00:00:00 GMT

In this weekly newsletter, I write a summary of posts that got my attention during the week. I center around topics AWS, DevOps, Architecture, AWS CDK, Data Engineering, Data Analytics and much more.

If you like my content follow me on my usual spots to hang around:

Improve DX by publishing an API SDK - a CDK Serverless example

https://www.rehanvdm.com/blog/improve-dx-publishing-an-api-sdk-cdk-serverless

Uii an API SDK is a nice idea. For me, that looks similar to OpenAPI / Swagger SDKs. It seems to be a smart combination of OpenAPI SDK and TypeScript. Plus with a sophisticated CI/CD flow like posting to slack you can make developers earlier aware of SDK changes. Interesting idea please read the full article!

Thanks a lot to the author Rehan van der Merwe

Getting started with testing DynamoDB code in Python

https://dev.to/aws-builders/getting-started-with-testing-dynamodb-code-in-python-dif

Interesting post about testing DynamoDB code in Python. You can use the python package moto to mock the AWS Service. That is super cool and how I used to when I am working with jest in TypeScript. Super cool that Python has something similar :). So I recommend reading the full article if you are interested in programming AWS CDK or AWS SDK with Python.

Thanks a lot to the author Maurice Borgmeier

How to work around CloudFormation circular dependencies

https://theburningmonk.com/2022/05/how-to-work-around-cloudformation-circular-dependencies/

Super interesting. CloudFormation circular dependencies are an often encountered problem when working with CloudFormation or with AWS CDK (which is basically CloudFormation). So this article is super useful for how to resolve those.

Thanks a lot to the Serverless Hero Yan Cui

Solutions Architect Tips: How to Build Your First Architecture Diagram

https://www.readysetcloud.io/blog/allen.helton/how-to-build-your-first-architecture-diagram/

An interesting article about architecture diagrams. It introduced and recommends the use of c4model which is super interesting https://c4model.com/#SystemContextDiagram .

Thanks a lot to the author Allen Helton

Introduction to AWS Developer Tools - Part 1

https://dev.to/kcdchennai/introduction-to-aws-developer-tools-part-1-590b

Super cool overview and summary of existing AWS Developer Tools. My highlight is cloud9. Look at the article if you want to know why.

Thanks a lot to the author MakendranG

Source Control your AWS CloudFormation templates with GitHub

https://dev.to/techielass/source-control-your-aws-cloudformation-templates-with-github-8im

This is a cool article about how to manage your CloudFormation template via GitHub Actions like linting and more. But yeah I would recommend not using CloudFormation directly and using AWS CDK :).

Thanks a lot to the author Sarah Lean

What exactly is Frontmatter?

https://dev.to/dailydevtips1/what-exactly-is-frontmatter-123g

Explains what Frontmatter is. I didn't realize that I already use it for my blog post Gatsby side martinmueller.dev :D.

Thanks a lot to the author Chris Bongers

Why CI/CD? How was a life without CI/CD & how is it now?

https://dev.to/wardaliaqat01/why-cicd-how-was-a-life-without-cicd-how-is-it-now-46bl

A good and comprehensive article on why CI/CD is so awesome. Use this article to sell CI/CD to whomever ^^.

Thanks a lot to the author Warda Liaqat

Security Iceberg: AWS Security Hub the right way

https://cloudonaut.io/security-iceberg-aws-security-hub-the-right-way/

A nice introduction and roundup about AWS Security Hub. Gives as well some tips when working with it.

Thanks a lot to the author Andreas Wittig

Docs for Everyone

https://dev.to/meganesulli/docs-for-everyone-28pm

Nice article about how to write meaningful documentation. Highlights the importance of diagrams.

Thanks a lot to the author Megan Sullivan

What makes a good DevOps manager?

https://www.reddit.com/r/devops/comments/ustzqj/what_makes_a_good_devops_manager

A great discussion about what makes a great DevOps Manager.

What Are AWS CDK Constructs, Stacks and How To Use Them

https://interweave.cloud/2022/05/17/what-are-aws-cdk-constructs-stacks-and-how-to-use-them/

A good explanation about AWS CDK level 1, 2 and 3 constructs.

Thanks a lot to the author Faizan Raza

Is This Thing On? Giving an Effective Talk In Person

https://dev.to/aws-builders/is-this-thing-on-giving-an-effective-talk-in-person-887

An interesting talk about public speaking. The author gives tips on how to improve public speaking.

Thanks a lot to the author Kristi Perreault

ESLint and Prettier

https://dev.to/suchintan/eslint-and-prettier-51e0

Nice post about why eslint and prettier are important.

Thanks a lot to the author SUCHINTAN DAS

Open Source Projects

In this section, I list Open Source projects I find super interesting!

json-schema-to-typescript

https://www.npmjs.com/package/json-schema-to-typescript

Import TypeScript type from JSON. That was always a disadvantage when a client wanted me to use a JSON file as input for CDK properties. It is super cool that it seems you easily can infer the type.

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 19

Sun, 15 May 2022 00:00:00 GMT

In this weekly newsletter, I write a summary of posts that got my attention during the week. I center around topics AWS, DevOps, Architecture, AWS CDK, Data Engineering, Data Analytics and much more.

If you like my content follow me on my usual spots to hang around:

Simple Leave Management System with AWS Serverless

https://dev.to/aws-builders/simple-leave-management-system-with-aws-serverless-3dnp

Constructing a workflow like the explained simple leave management with AWS Step Functions is so cool. Furthermore, the recently released AWS SDK service integrations makes it fun to define such.

Thanks a lot to the author Pubudu Jayawardana

AWS open-source news and updates, #112

https://dev.to/aws/aws-open-source-news-and-updates-112-5bjp

And again a nice weekly summary. My highlight is the aws-dataall as it looks super interesting for sharing ML data company internally and event externally mhh. Lots to think about here. But as always there is tons of other gold so you have to explore that!

Thanks again to the author Ricardo Sueiras. I hope you enjoyed the Summit in Berlin ^^. Next time count me in!

Simple Serverless Scheduler

https://dev.to/aws-builders/simple-serverless-scheduler-25c7

A cool article for how to solve an interesting business case like when a user cancels their subscription but still needs to have access until the end of the paying period time. The author compared different possible solutions for how to implement such a serverless scheduler.

Thanks a lot to the author Pawel Zubkiewicz

Debugging AWS Step Functions executions with the new console experience

https://aws.amazon.com/blogs/compute/debugging-aws-step-functions-executions-with-the-new-console-experience/

The capability that you can debug Step Functions became super important to me and this article is great to learn more about it. The new debug AWS Console experience for Step Functions looks super exciting. Can't wait to try it myself!

Thanks a lot to the author Benjamin Smith.

Serverless Threat Modelling Part 2 :rocket:

https://leejamesgilmore.medium.com/serverless-threat-modelling-part-2-87e69d447fa8

A nice top-down post on serverless threat modeling.

Thanks a lot to the author Lee James Gilmore

Architectural Patterns

https://dev.to/dev0928/architectural-patterns-5f31

Nice roundup of the architectural patterns Monolith Architecture, N-tier Architecture, Service-oriented Architecture, Message-based Architecture, Microservice-based Architecture, Reactive Architecture, and Serverless Architecture.

Thanks a lot to the author dev0928

Redshift vs Athena

https://www.reddit.com/r/aws/comments/umx793/redshift_vs_athena/

Interesting discussion about AWS Redshift vs Athena. Athena seems to keep the costs low as it is serverless but it would be rather slow when you need a quick answering kind of API.

3 Reasons You Should Stay Away From AWS Step Functions

https://www.readysetcloud.io/blog/allen.helton/when-not-to-use-step-functions/

Very nice deep dive into AWS Step Functions. It is so cool that the author describes some problems and how to solve them from a view as a Step Functions Beginner, Intermediate and Advanced. Nice idea! Must read!

Thanks a lot to the author Allen Helton

Annahita Esmailzadeh: „Ich entspreche nicht dem Stereotyp eines IT-lers“

https://t3n.de/news/annahita-esmailzadeh-stereotype-it-tech-branche-1471342/?utm_source=rss&utm_medium=feed&utm_campaign=news

Super interesting talk about fewer women in IT and how hard it is for women to be acknowledged in the IT area. Unlucky it is in German but you can use a translation service :).

Thanks a lot to the author Andreas Weck

On DevOps: 1. What It Is

https://dev.to/devstream/on-devops-1-what-it-is-3lmd

Interesting discussion about DevOps and DevOps Culture. Like what it is and more. As well as in this context what does leadership mean.

Thanks a lot to the author Tiexin Guo

Starter Guide & “Things I wish I knew before” for AWS Lambda

https://towardsaws.com/starter-guide-things-i-wish-i-knew-before-for-aws-lambda-fe36ce2fd4cb

A nice deep dive into AWS Lambda regarding pricing, when to use it, limitations and much more. It even tackles the cold start problem and gives good advice on how to minimize that burden. Super cool!

Thanks a lot to the author Tobias Schmidt

Automating machine learning lifecycle with AWS

https://trojrobert.medium.com/automating-machine-learning-lifecycle-on-aws-trojrobert-385a66b82fbf

A very broad listing of useful AWS services related to machine learning. Though I don't believe that it shall only take 11 minutes to read :D !!

Thanks a lot to the author Robert John

Shortcut for AWS CDK credentials: insanely simple setup for SSO, SAML, and named profiles

https://dev.to/aws-builders/shortcut-for-aws-cdk-credentials-insanely-simple-setup-for-sso-saml-and-named-profiles-35a9

An interesting post on how to use SSO SAML and named Profiles for AWS CDK deployments. TBH I never used CDK deploy --profile $PROFILE_NAME but yeah if I have to this article explains it in very good detail.

Thanks a lot to the author Alessandro Gaggia

How to get remote software developer jobs in 2022 | high paying | international | remote job

https://www.youtube.com/watch?v=ftC-jQgmcwI

Contains very good tips for how to get a remote developer job.

Thanks a lot to the author Learn And Grow Everyday

Securing AWS Lambda function URLs

https://www.wiz.io/blog/securing-aws-lambda-function-urls

Nice roundup about what are vulnerabilities from AWS Lambda Function Urls

Thanks a lot to the author Amitai Cohen

AWS Security Best Practices with Crowd Sentry

https://blog.crowdsentry.io/aws-security-best-practices-with-crowd-sentry-b3605f1854bc

A nice short summary about some AWS Security Best Practices and how to keep those.

Thanks a lot to the author Mike Wade

Reamplify

https://github.com/thomasrooney/reamplify

An Open Source CDK Community project which I find super interesting. It is doing cherry-picking from AWS Amplify UI and AWS CDK for deployment. I do that in my private projects as well for example https://github.com/senjuns/senjuns. I think the author could enhance/simplify its repo even more by using https://github.com/projen/projen for the project setup.

Thanks a lot to the author Thomas Rooney

Securely Use Secrets In Your AWS CDK Deployed Lambda Function

https://dev.to/wesleycheek/securely-use-secrets-in-your-aws-cdk-deployed-lambda-function-1ebb

Explains how to use AWS Secrets Manager for storing secrets related to your CDK deployments. And the example is written in Python :).

Thanks a lot to the author Wesley Cheek

How Pair Programming Helps The Team

https://dev.to/michaelprosario/how-pair-programming-helps-the-team-3045

I find the topic of pair programming super interesting. I never had the opportunity to do that but I will change that! This article is a very good roundup of why pair programming is nice. As well it gives details on how to do pair programming.

Thanks a lot to the author Michael Rosario

How to become a good developer?

https://dev.to/this-is-learning/how-to-become-a-good-developer-7no

Speaks to the bottom of my heart.

Thanks a lot to the author MD Sarfaraj

Serverless apps: Why IaC should be the direction

https://zachjonesnoel.hashnode.dev/serverless-apps-why-iac-should-be-the-direction

A nice article on why to use Infrastructure as Code (IAC).

Thanks a lot to the author Jones Zachariah Noel N

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Create and delete AWS Accounts programmatically

Wed, 11 May 2022 00:00:00 GMT

Hi.

In AWS, it is best practice to create new accounts if, for example, you want to deploy different stages such as dev, qa and prod for your solution. It is also common to create so-called sandbox AWS accounts to provide these new employees for testing AWS services. There are many more reasons why new AWS accounts should be created.

Now creating such AWS accounts can be quite time consuming if it has to be done manually. With AWS Organizations, the AWS JavaScript SDK and a little TypeScript knowledge, the process can be automated for the most part. In this post, I'll introduce how this works. To do this, I first explain what AWS Organizations is in the first place and then I introduce the two TypeScript scripts that can create and delete the AWS Sandbox accounts using the AWS SDK.

AWS Organizations

AWS Organizations helps to manage the AWS accounts. In so called Organizational Unites (OU) accounts can be grouped together. Furthermore, security mechanisms such as policies can be defined across accounts. AWS Organizations offers much more functionality and I would like to refer to the public documentation. Interesting for this post is that there is also a JavaScript SDK wrapper for the AWS Organizations API that we will use in the next sections to dynamically create and delete accounts.

createAccount and deleteAccount

I created a small helper library in GitHub to simplify the programmatic creation and deletion of AWS accounts. An AWS SDK script could then be easily built such as:

import {
 createAccount,
 moveAccountToOU,
 getAccountIdFromName,
} from "aws-accounts

export async function main(): Promise<void> {
 var args = process.argv.slice(2)

 if (args.length !== 1) {
  usage()
  throw new Error("Wrong number of arguments")
 }

 const accountName = args[0]

 const response = await createAccount(accountName)
 const accountId = await getAccountIdFromName(accountName)
 if (accountId) {
  await moveAccountToOU(accountId, "ou-zblx-w7yw0qge")
 }
}

The code can also be seen in the repo here.

The library is a lightweight wrapper around AWS SDK TypeScript v.2 . With await createAccount(accountName) the account is created first. Then with await getAccountIdFromName(accountName) and await moveAccountToOU(accountId) the account is moved to the defined OU (Organizational Unit).

Summary

In this blog post I explained why it is sometimes necessary to create new AWS accounts and how this can be done using AWS Organizations and the JavaScript SDK. For this I created a createAccount and a deleteAccount script and presented them here. Did you find the post helpful or do you have any suggestions for improvement? Let me know and write me :).

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

Newsletter martinmueller.dev 2022 week 18

Sun, 08 May 2022 00:00:00 GMT

In this weekly newsletter, I write a summary of posts that got my attention during the week. I center around topics AWS, DevOps, Architecture, AWS CDK, Data Engineering, Data Analytics and much more.

If you like my content follow me on my usual spots to hang around:

Getting started with Git and Github

https://dev.to/mbugua70/getting-started-with-git-and-github-3mg

Good roundup for what is Git and the most important commands. It would help you to get more familar with the Git CLI which make me for example very productive. Furthermore I recommend setting this alias:

alias ginit="git add --all && git commit -m "init" && git push"

gaddall() { git add --all && git commit -m "$1" && git push; }
alias gall="gaddall"

So with that I just need line ginit or gall for quickly adding the commit. That was a super productive boost for me.

17 Ridiculously Funny Programming Memes that Every Developer Can Relate To

https://dev.to/muthuannamalai12/17-ridiculously-funny-programming-memes-that-every-developer-can-relate-to-2gg3

A very good brain relaxes and it is super funny.

AWS Step Functions 101

https://dashbird.io/blog/ultimate-guide-aws-step-functions/

A good roundup about AWS Step Functions. It is very good if you want to dive deep into the topic. As I have a lot with Step Functions already the article is a bit long. But yeah a highlight really is that you can use typed AWS CDK to define your Step Functions. Thanks to the author Mariliis Retter.

AWS open-source news and updates #111

https://dev.to/aws/aws-open-source-news-and-updates-111-1dj3

Here again where Ricardo presents exciting open source community projects. My favorites once are:

The cloudfront-manager. Juggling CloudFront distribution in AWS is always not easy regarding Invalidate, Enable, Disable and Delete. This tool looks very promising in helping you with that! https://dev.to/aws-builders/aws-cloudfront-manager-a-windows-utility-for-cloudfront-17o0 . Thanks a lot to the author Harinder Seera
amazon-cloudwatch-retention-period-setter. This tool help you set the log retention period for CloudWatch logs per default. Otherwise, they would stay in CloudWatch forever and accumulate costs. I usually prefer setting AWS CDK with the right properties to avoid that but yeah very often, I am lacy doing that so having something more globally like that would be awesome. https://aws.amazon.com/blogs/infrastructure-and-automation/reduce-log-storage-costs-by-automating-retention-settings-in-amazon-cloudwatch/. Thanks to author Mohamed Wali.

And finally thank you so much Ricardo for picking and presenting all those cherries I missed this week. And dear reader please support Ricardo Sueiras! Man, I would love to meet you at AWS Summit Berlin but I have to stay behind here in south Portugal Albufeira on the beach during my holidays. Not that bad actually :P.

Obsidian - An IDE for your Brain

https://dev.to/envoy_/obsidian-an-ide-for-your-brain-1bn7

I used Obsidian for some weeks and is really cool. But I figured that VS Code is kind of better when it comes to writing / organizing my content. Probably combining both like VS Code for writing and Obsidian for viewing seems interesting.

Optimize your workloads for Sustainability

https://globaldatanet.com/tech-blog/optimize-your-workloads-for-sustainability

Super good roundup about sustainability with AWS. Explains the shared responsibility between AWS and customers and provides some very useful tips for how to reduce the resource footprint. Thanks to the author David Krohn.

:clown_face: AWS CDK 101 - 🤾‍♂ Using layers in Lambda functions and saving JSON to S3

https://dev.to/aravindvcyber/aws-cdk-101-using-layers-in-lambda-functions-and-saving-json-to-s3-46fg

Super interesting article about how and why to use Lamba Layers. TBH I always avoided them but that might change because of this post. Thanks to Aravind V.

Post Messages To Slack Using AWS Lambda Function URLs

https://dev.to/josuebustos/post-messages-to-slack-using-aws-lambda-function-urls-5f2a

A nice in-depth article about how to post messages to slack using AWS Function URLs. Would be even cooler to create an AWS CDK construct to simplify that. Thanks to the author Josue Bustos

Website to PDF using AWS Lambda Function URLs

https://jobinbasani.com/2022/05/03/website-to-pdf-using-aws-lambda-function-urls/

In this article, the author creates via AWS CDK and JS library chrome-aws-lambda a Lambda which can transform a website into PDF. Very cool. Thanks to the author Jobi Basani

Terraform vs Pulumi vs Cloud SDKs - Keep It Simple, Silly

https://www.youtube.com/watch?v=Z3hyqmCddXc

This topic is super hot. Thanks to the creator Michael Crilly

How to avoid AWS unintentional charges?

https://www.internetkatta.com/how-to-avoid-aws-unintentional-charges

Recommend using "AWS Pricing Calculator" to evaluate your pricing model.

Programmieren mit Python: Große Datenmengen verwalten mit vaex

Programming with Python: How to manage a huge amount of data with vaex

https://www.heise.de/ratgeber/Python-Mit-vaex-grosse-Datenmengen-verwalten-7066766.html

With Python Panda you can not process your data anymore when they become too big. The library vaex is solving that problem. Thanks to the author odi

Pair Programming with a senior is invaluable

https://dev.to/w3ndo/pair-programming-with-a-senior-is-invaluable-2fdo

For me, that topic is super important. I totally agree pair programming is invaluable. We have an unhealthy ratio between junior and senior engineers. So pair programming really helps to fix that. And in my opinion, it is super helpful for the senior as well.

TypeScript vs JavaScript: What's the difference?

https://dev.to/educative/typescript-vs-javascript-whats-the-difference-n5m

Super good summary of what is JavaScript and what TypeScript is and why to favor one over the other. In the future, I will refer just to this article.

Thanks a lot to Hunter Johnson

Infrastructure-as-Code: Vierte Pulumi-Hauptversion verspricht Universal IaC

Translated: Infrastructure-as-Code: Fourth Pulumi-version promises universal IaC

https://www.heise.de/news/Infrastructure-as-Code-Vierte-Pulumi-Hauptversion-verspricht-Universal-IaC-7076433.html

Version 4 from Pulumi seems to make an interesting move by incorporating AWS CDK.

Thanks a lot to the author map

Cloud Pricing Comparison: AWS vs. Azure vs. Google Cloud Platform in 2022

https://dev.to/castai/cloud-pricing-comparison-aws-vs-azure-vs-google-cloud-platform-in-2022-no2

Interesting comparison between the three big cloud providers. In the article, they use similar VMs from different providers to compare those costs. That is a smart idea! As well they compared spot instances from the different providers.

Thanks a lot to the author CAST AI

Developer shortage? No, just bad interviews

https://dev.to/jssantana/developer-shortage-no-just-bad-interviews-1cpa

The author explains why most companies are hiring wrong. Just those whiteboard kinds of tests don't evaluate the skills of the developer. I totally agree and those kinds of tests always bothered me.

Thanks a lot to the author Jean Santana

Using Athena to query multi-account Cloudwatch Logs

https://dev.to/markymarkus/using-athena-to-query-multi-account-cloudwatch-logs-54j

The author explains how you can store your CloudWatch logs via Kinesis in S3 and how you can query them. That is super cool and a preferred way when logging gets more complex! And I like that the author used a diagram to explain the bit complex architecture.

Thanks a lot to the author markymarkus

Does AWS Lambda good for ETL jobs?

https://dev.to/aws-builders/does-aws-lambda-good-for-etl-jobs-10i8

The author discusses if Lambda is well suited for ETL. I think it is until you reach a Lambda limit and then you change to other services like AWS Glue.

Thanks a lot to the author Mohamed Latfalla

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 17

Sun, 01 May 2022 00:00:00 GMT

In this weekly newsletter, I write a summary of posts that got my attention during the week. I center around topics AWS, DevOps, Architecture, AWS CDK, Data Engineering, Data Analytics and much more.

If you like my content follow me on my usual spots to hang around:

Using Obsidian

https://dev.to/paratron/introducing-and-using-obsidian-3o17

I use and love Obsidian already. It is a great tool to organize content extracted from posts, YouTube and more. And most cool of it is based on Markdown <3. I encourage you to try it!

Update: I figured that VS Code is probably as good as Obsidian for managing your content. Even better you can open multiple tabs in VS Code which you can't in Obsidian.

What is WebAssembly and why it is a game-changer?

https://dev.to/aryank21/what-is-webassembly-and-why-it-is-a-game-changer-21p4

Webassembly or short WASM is super interesting. It advertises developers with execution speed compared to js solutions. For a greenfield project, I definitely want to try it. Super interesting as well is to get potentially closer to the hardware like perhaps IoT devices.

Zero Cold Start for Lambda

https://www.reddit.com/r/aws/comments/ud5fos/zero_cold_start_for_lambda/

Super interesting discussion about zero cold starts for Lambda. That highlights the big conflict like serverless vs zero cold start Lambda. Zero cold starts are possible but not without trade-offs. Sure the customer wants that without trade-offs. That is really hard.

How to speed up Lambda Functions

https://www.freecodecamp.org/news/how-to-speed-up-lambda-functions/

So Lambda hast different phases like init, invoke and shutdown. Initialization code should be put into the init phase which means outside of the lambda handler function.

EKS Blueprints: IaC Modules for Production-Ready Kubernetes

https://itnext.io/eks-blueprints-iac-modules-for-production-ready-kubernetes-48032d5ce88#2ee6-c02bad69ddca

Super summarizing about the new-ish AWS CDK EKS Blueprints. That topic seems very hot and very promising to be a nice abstraction on top of K8S. Unlucky I am more on the serverless-ish side so, I probably miss some cherries atm.

Deploy Serverless Containerized NodeJs Apps on AWS ECS Fargate with AWS Copilot

https://www.serverlessguru.com/blog/deploy-serverless-containerized-nodejs-application-on-aws-ecs-fargate

Good roundup about what is AWS ECS, Fargate and Copilot. It made me curious to learn more about Copilot as it seems to have some cool CI/CD capabilities.

Websocket with React

https://dev.to/koladev/websocket-with-react-nodejs-and-docker-building-a-chat-application-3447

Create a React-based chat using WebSockets. Uses Tailwindcss which is cool.

5 Useful TypeScript Features In VS Code

https://dev.to/danielfy/5-useful-typescript-features-in-vs-code-4am

Wow that will enhance my developing velocity O.O. Huge thanks to https://twitter.com/danielfyhr

Input on my study path to step into DevOps

https://www.reddit.com/r/devops/comments/uarh7i/input_on_my_study_path_to_step_into_devops/

Nice advice for a DevOps Junior.

AWS CDK v2.21.0 Release Highlights

https://www.youtube.com/watch?v=GxXKJDYnpDY

Release notes highlights from AWS CDK 2.21.0 from AWS Twitter presented from the AWS Hero @mattbonig.

AWS CDK v2.22.0 Release Highlights

https://www.youtube.com/watch?v=EyLFDt6uRiI

AWS Amplify Studio

https://dev.to/aspittel/aws-amplify-studio-is-generally-available-5-new-features-to-checkout-4h0m

AWS Amplify Studio is super interesting but I still miss some features like generating React TypeScript and Styled Components.

Using Amplify UI Builder with your existing data

https://dev.to/codebeast/using-amplify-studio-with-your-existing-data-5a1n

Nice description of how to use Amplify Studio.

Writing RFCs

https://dev.to/wesen/quick-tip-tuesday-writing-rfcs-for-fun-and-profit-3bo

Explains how important RFCs are

Using Apollo Server on AWS Lambda with Amazon EventBridge for real-time, event-driven streaming

https://aws.amazon.com/blogs/opensource/using-apollo-server-on-aws-lambda-with-amazon-eventbridge-for-real-time-event-driven-streaming/

Amazing post about Apollo GraphQL and AppSync. Very cool is that it recommends CDK to do the infrastructure.

Women IT specialists

https://www.heise.de/news/IT-Spezialistinnen-dringend-gesucht-Unternehmen-hoffen-auf-mehr-Frauen-7066656.html

Great topic about how to get more women in tech. It seems it would be useful to provide young girls earlier with information about IT jobs.

# How to control access to AWS resources based on AWS account, OU, or organization

https://aws.amazon.com/blogs/security/how-to-control-access-to-aws-resources-based-on-aws-account-ou-or-organization/

Explains how the new IAM condition key can be used for more fine granular set policies.

How to Load Test Your Apps For Free By Going Serverless

https://dev.to/aws-builders/how-to-load-test-your-apps-for-free-by-going-serverless-1cl2

A very cool top-down description for how to load test your serverless deployments.

Is DevOps in my environment possible?

https://www.reddit.com/r/devops/comments/udj8s8/is_devops_in_my_environment_possible/

Pretty cool discussion about how to introduce and what is DevOps

How to configure ESLint, Prettier, Husky, Lint-staged into a React project with TypeScript and Tailwind CSS

https://dev.to/ixartz/how-to-configure-eslint-prettier-husky-lint-staged-into-a-react-project-with-typescript-and-tailwind-css-4jp8

Nice how-to for bringing those low-hanging fruits like eslint, prettier, husky and lint staged into your React project. I use lots of that in https://github/senjuns/senjuns.

Exploitation and Prevention of common AWS Vulnerabilities

https://csaju.com/blog/exploitation-and-prevention-of-common-aws-vulnerabilities/

Give a nice roundtrip for how to prevent exploitation on the AWS services S3 and Lambda.

Amazon Rekognition introduces Streaming Video Events to provide real-time alerts on live video streams

https://aws.amazon.com/blogs/machine-learning/amazon-rekognition-introduces-streaming-video-events-to-provide-real-time-alerts-on-live-video-streams/

Helping users to AI surveillance their backyard to protect their home. Aims for minimizing the false alerts.

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot of my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on:

And don't forget to visit my site

Newsletter martinmueller.dev 2022 week 16

Mon, 25 Apr 2022 00:00:00 GMT

In this weekly newsletter, I write a summary of posts that got my attention during the week. I center around topics AWS, DevOps, Architecture, AWS CDK, Data Engineering, Data Analytics and much more.

If you like my content follow me on my usual spots to hang around:

Self-healing

https://dev.to/jobber/self-healing-code-46o9

Self-healing code with graceful degradation. Very cool idea that the solution keeps running with limited functionality even when a large portion of it doesn't work properly. There should be/are plenty of examples where Netflix or Amazon is doing stuff like that. The author even takes it a step further like having a read-replica database to keep serving HTTP GET reads but blocking writes when the main database is down and an HTTP GET write is requested. That approach sounds very promising in general and probably can be transferred to other similar situations as well.

Fuck-up-Stories

https://t3n.de/news/fuckups-fehler-fehlerkultur-fehlermanagment-1459624

Interesting content about Fuck-up-Stories in Companies and how healthy they can be when doing in a company. In the long term, it can increase the revenue of a company.

Jaeger QuickStart

https://betterprogramming.pub/introducing-jaeger-quick-start-deploying-on-aws-f0ee5d398e8a A post about Jaeger and how to run it in a QuickStart. Interesting as I didn't work much with Jaeger so much. It highlights how cool such Quick Starts are for starting quick :). Very interesting seems to be the fact that you can combine Jaeger with AWS X-Ray.

Joining DevOps

https://www.reddit.com/r/devops/comments/u7p6w9/joining_devops/

A new DevOps Junior after getting a DevOps-ish Job asking for help with where to start in the DevOps.

Lambda with AWS CDK and EFS

https://dev.to/wesleycheek/lambda-function-with-persistent-file-store-using-aws-cdk-and-aws-efs-45h8

Deploy AWS Lambda with CDK and EFS attached to it.

AWS Well-Architected - Security Pillar

https://dev.to/sebastiantorres86/aws-well-architected-framework-security-pillar-2jhc

A nice summary whey security and what is important regarding AWS Security. Have a nice best practice list.

API Design - GraphQL vs Rest

https://dev.to/wundergraph/api-design-best-practices-for-long-running-operations-graphql-vs-rest-1mkb

Has an interesting example of how asynchronous API can be done asynchronous API with GraphQL and REST API. Then the author compares the GraphQL and REST API solutions with each other.

AWS Step Function scheduler for EKS nodes

https://www.automat-it.com/post/custom-kubernetes-scheduler-with-eks-and-step-functions-for-machine-learning-workloads

Cool solution for how to start EKS nodes when they are needed with Step Function.

Amazon DevOps Guru for serverless

https://aws.amazon.com/blogs/aws/automatically-detect-operational-issues-in-lambda-functions-with-amazon-devops-guru-for-serverless/

DevOps Guru for Serverless seems super interesting for like optimizing your serverless deployments

AWS Organizations and AWS SSO

https://www.reddit.com/r/aws/comments/u9niif/aws_organizations_and_aws_sso/

A Reddit question for how to set up AWS Organizations and AWS SSO. I recommended giving https://github.com/superwerker/superwerker a try.

VPN tunnel with AWS

https://www.thenetworkdna.com/2022/01/site-to-site-vpn-tunnel-from-aws-to.html

VPN tunneling is already a tough topic at least for me. Cool instructions but I would prefer to have it as code somehow.

AWS Policy as Code

https://dev.to/aws-builders/introduction-to-policy-as-code-5bma

Policy as Code is getting more and more attraction. As an AWS CDK fanboy, I guess I do that for many years now ^^.

Software Architecture and Abstraction

https://dev.to/vickodev/software-architecture-and-abstraction-for-newbie-3nob

Very interesting post about what is a software architecture, software design patterns and abstractions.

Final Words

Thank you to the authors of those amazing posts. And thank you to the readers of the newsletter. When you like this format or know how to improve it please let me know :). Let's Build!

I love to work on Open Source projects. A lot from my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on Patreon:

Export DynamoDB Items to S3

Sun, 03 Apr 2022 00:00:00 GMT

Storing data like JSON logs in DynamoDB is a great idea as DynamoDB is very scalable. In addition, it is easier to transfer data into a DynamoDB table using for example Lambda and AWS SDK. Also, it makes analyzing the logs easier for example the AWS Console offers great filtering options to search for specific so-called table items.

This all sounds very good but there is one hitch and that is the cost. As the number of items increases, so does the cost. So it would be advisable to delete the DynamoDB data from the table after a certain time, e.g. 30 days, and import it into an S3. The costs for S3 are much lower and it would even be possible to reduce them if you use a cheaper S3 tier like Glacier.

The question now is how you can do this most sophisticated? For me the combination DynamoDB Time To Live, DynamoDB Streams and Lambda which writes to an S3 bucket worked very well. In the following section, I will describe my solution.

Solution

You can find the code in my GitHub repo https://github.com/mmuller88/cdk-ddb-to-s3 . As you can see the architecture in the title image is simple. DynamoDB Streams invokes a Lambda, which writes the deleted item away to S3. In my example, the DynamoDB items are JSON logs with few properties. In your case, the DynamoDB item can look different. But the basic concept should still the same!

By the way, if you are curious how I created the AWS Component Architecture Diagram, you can read it here.

const table = new ddb.Table(this, "table", {
 partitionKey: { name: "systemId", type: ddb.AttributeType.NUMBER },
 sortKey: { name: "timestamp", type: ddb.AttributeType.NUMBER },
 stream: ddb.StreamViewType.OLD_IMAGE,
 timeToLiveAttribute: "ttl",
})

const ddbArchiveBucket = new s3.Bucket(this, "ddbArchiveBucket")

const ddbArchiveLambda = new lambdajs.NodejsFunction(this, "ddbArchiveLambda", {
 reservedConcurrentExecutions: 1,
 environment: {
  DDB_ARCHIVE_BUCKET_NAME: ddbArchiveBucket.bucketName,
 },
})
table.grantStreamRead(ddbArchiveLambda)
spyderArchiveLogBucket.grantWrite(ddbArchiveLambda)

ddbArchiveBucket.addEventSourceMapping("archivelog", {
 // max json document is 4 mb per file
 batchSize: 10000,
 maxBatchingWindow: core.Duration.minutes(5),
 eventSourceArn: table.tableStreamArn,
 startingPosition: lambda.StartingPosition.TRIM_HORIZON,
 bisectBatchOnError: true,
})

stream: ddb.StreamViewType.OLD_IMAGE is set because I am only interested in the items that have been deleted.

In the table definition a TimeToLive attribute was set with timeToLiveAttribute: 'ttl' . It is then important that your component which writes the data to DynamoDB always sets the ttl attribute. This is a timestamp that describes when the item should be deleted. In my project this was after 30 days.

Usually, you should only need one running Lambda to write the data away to S3 reservedConcurrentExecutions: 1. That serves as an additional safeguard against many unexpected invocations of Lambda.

The addEventSourceMapping part is probably the most exciting. What is pretty cool is that DynamoDB Streams provides a batching feature. The Lambda can then process the deleted items as a batch. That reduces the number of Lambda calls and therefore the costs. The default DynamoDB default batching is not quite ideal for our use case here so I used AWS Console and the Lambda call metrics to optimize it:

batchSize: 10000,
maxBatchingWindow: core.Duration.minutes(5),

The batchSize with 10000 and the maxBatchingWindow are chosen maximally to call a Lambda really only every 5 minutes:

Super cool or? So the number of Lambda calls was massively reduced.

By the way, choosing the batch size with 10000 is ok in my case because they are JSON items and we stay under the maximum size of 4 MB per file if I write them away to S3. In your case this batch size could be to high. You have to try this out with testing. I just deleted 300 log items at once and it was still successfully written to S3 with a Lambda call.

With Athena and SQL queries the data in the S3 bucket can be inspected. How exactly that works I do not want to go into here. There are great tutorials that explain how Athena can be used to query for specific information in the S3 data such as JSON files.

Also I would like to mention that in some tutorials I have also seen the use of Kinesis Firehose. But I don't think you need it for your solution. DynamoDB Streams already has the ability to batch. But maybe it is another reason why Firehose was used. If you know why Kinesis Firehose might be useful write me!

Summary

Writing DynamoDB items away to S3 to save costs is super cool. Here I explained how you do it. Do you have any feedback on this post or other suggestions on what I can cover? Then let me know!

I love to work on Content Management Open Source projects. A lot from my stuff you can already use on https://github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on Patreon:

Become a Patreon!

AWS CDK diagrams with cdk-dia <3

Thu, 24 Mar 2022 00:00:00 GMT

Hi.

AWS component diagrams are a great way to visualize complex AWS architectures. For many, they are much more intuitive than, for example, CDK code. I myself work a lot with AWS component diagrams when I develop new AWS systems. What has always bothered me is that creating the diagrams is time consuming and they have to be updated when the architecture changes.

If you feel the same way, try cdk-dia. I used it in my last two CDK projects and the experience was simply overwhelming.

How does it work?

The tool cdk-dia allows me to create AWS component diagrams automatically from my CDK code. It even follows the CDK Construct Level 1 and 2 abstractions. This means that e.g. the big components are symbols level 2 constructs and only the main component which usually gets Resource as ID. Thus, only the really important AWS components are rendered. This is also known as collapsing.

(collapsed diagram)

If you want to prevent collapsing, so that all underlying components are displayed, you can define a decorater in the code. How exactly this works you can see here .

(non-collapsed diagram)

This is what I used for my private asset bucket construct for example:

...
@DiagramOptions({ collapse: CollapseTypes.FORCE_NON_COLLAPSE })
export class PrivateAssetBucket extends core.Construct {
...

If you want to know more about my Prowler Construct look here.

Comparison with AWS Console Cloudformation template

The AWS Console itself also has a diagram tool to visualize CloudFormation stacks called AWS CloudFormation Designer. You can find more information here. So I compared the designer with cdk-dia and I think cdk-dia is a lot better. cdk-dia manages to achieve a much better level of abstraction through collapsing. Also the graphics of cdk-dia are much nicer.

In the following section I explain how you can implement cdk-dia in your CDK project.

cdk-dia script

For continuous updating of the cdk-dia diagrams it would be a good idea to write a small helper script in the package.json. But first you should make sure that cdk-dia is installed in the dev dependency scope with yarn add cdk-dia -D.

Then create a folder named diagrams and add this script to package.json.

"scripts": {
    ...
    "dia": "yarn cdk synth && yarn cdk-dia && mv diagram.png diagrams/dashboard.png",
    ...
  },

The script first synthesizes the current CloudFormation templates. cdk-dia needs them to generate the diagram at all. Then yarn cdk-dia creates the diagram and moves it to the diagrams folder.

yarn cdk-dia also creates a diagram.dot file. I recommend to put the following two files into the .gitignore:

diagram.dot
diagram.png

If you like to split the diagram into single images, you can also render only single CDK stacks, e.g.

"scripts": {
    ...
    "dia": "yarn cdk synth && yarn cdk-dia --stacks DashboardAppStack DashboardBackendStack && mv diagram.png diagrams/dashboard.png && yarn cdk-dia --stacks LandingPageStack && mv diagram.png diagrams/landingpage.png",
    ...
  },

Also a good idea is to integrate the cdk-dia command into the cdk synth process. I included this for example in my cdk Prowler Construct. In the package.json you will find the following script:

"scripts": {
    ...
    "synth": "yarn cdk synth && yarn cdk-dia && mv diagram.png diagrams/prowler.png",
    ...
  },

Now every time you run cdk synth manually, the cdk-dia diagram is rendered as well. Cool or? If you want to know more about my Prowler Construct look here.

Ideas

In the previous post I introduced a small script that generates the cdk-dia diagrams. Unfortunately this script still has to be run manually. It would be mega cool if these diagrams update themselves quasi. This should be possible if you use Husky pre-commit hooks. I am currently trying this out. If you want to know more about it, let me know.

Unfortunately cdk-dia couldn't render my CDK CD staging pipeline properly and I had to create it by hand in https://draw.io. It would be super cool if cdk-dia could also map pipelines.

Summary

If you also like to work with AWS component diagrams give cdk-dia a try! Here in the post I described how great the tool is and how easy it is to use. Feel free to contact me if you have any questions.

Thanks to the DeepL translater (free version) for helping with translating to English and saving me tons of time :).

To the wonderful readers of this article, I'm saying that feedback of any kind is welcome. In the future, I will try to include a discussion and comment feature here. In the meantime, please feel free to send me feedback via my social media accounts such as Twitter or FaceBook. Thank you very much :).

Become a Patreon!

Deploy Lambda@Edge with AWS CDK and TypeScript

Sun, 06 Feb 2022 00:00:00 GMT

Hi,

Lambda@Edge are Lambdas that are executed in the AWS Cloudfront Locations. That can increase performance and reduce latency compared to normal Lambda. Cloudfront together with Lambda@Edge can also being used as a proxy for a private S3 asset bucket.

A token issued by Cognito is then used by Lambda@Edge to validate if the requester is allowed to access the S3 asset. I will describe more about this in the next blogpost. Here we will talk about how to create a Lambda@Edge with AWS CDK and Typescript.

AWS CDK Lambda@Edge

Lambda@Edge already exists in the CDK cloudfront package https://docs.aws.amazon.com/cdk/api/v1/docs/aws-cloudfront-readme.html. Unfortunately, it is not directly possible to write the Lambda code with TypeScript as we are already used to it like from the NodejsFunction.

Well not with me! I took a closer look at NodejsFunction and EdgeFunction and transferred the TypeScript functionality from NodejsFunction to EdgeFunction. I simply define a CDK construct which is extended by EdgeFunction and take functions and properties from NodejsFunction. And here is the code of my nodejs-edge-function.ts construct:

import * as cloudfront from '@aws-cdk/aws-cloudfront';
import * as core from '@aws-cdk/core';
import * as bundling from '@aws-cdk/aws-lambda-nodejs/lib/bundling';
import * as lambda from '@aws-cdk/aws-lambda';
import * as fs from 'fs';
import * as path from 'path';
import { BundlingOptions } from '@aws-cdk/aws-lambda-nodejs';
import { PackageManager } from '@aws-cdk/aws-lambda-nodejs/lib/package-manager';
import { callsites, findUpMultiple } from '@aws-cdk/aws-lambda-nodejs/lib/util';


/**
 * environment variables are not supported for Lambda@Edge
 */
export interface NodejsEdgeFunctionProps extends Omit<lambda.FunctionOptions, 'environment'> {
    /**
   * Path to the entry file (JavaScript or TypeScript).
   *
   * @default - Derived from the name of the defining file and the construct's id.
   * If the `NodejsFunction` is defined in `stack.ts` with `my-handler` as id
   * (`new NodejsFunction(this, 'my-handler')`), the construct will look at `stack.my-handler.ts`
   * and `stack.my-handler.js`.
   */
    readonly entry?: string;

    /**
     * The name of the exported handler in the entry file.
     *
     * @default handler
     */
    readonly handler?: string;

    /**
     * The runtime environment. Only runtimes of the Node.js family are
     * supported.
     *
     * @default Runtime.NODEJS_14_X
     */
    readonly runtime?: lambda.Runtime;

    /**
 * Whether to automatically reuse TCP connections when working with the AWS
 * SDK for JavaScript.
 *
 * This sets the `AWS_NODEJS_CONNECTION_REUSE_ENABLED` environment variable
 * to `1`.
 *
 * @see https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/node-reusing-connections.html
 *
 * @default true
 */
    readonly awsSdkConnectionReuse?: boolean;

    /**
     * The path to the dependencies lock file (`yarn.lock` or `package-lock.json`).
     *
     * This will be used as the source for the volume mounted in the Docker
     * container.
     *
     * Modules specified in `nodeModules` will be installed using the right
     * installer (`npm` or `yarn`) along with this lock file.
     *
     * @default - the path is found by walking up parent directories searching for
     *   a `yarn.lock` or `package-lock.json` file
     */
    readonly depsLockFilePath?: string;

    /**
     * Bundling options
     *
     * @default - use default bundling options: no minify, no sourcemap, all
     *   modules are bundled.
     */
    readonly bundling?: BundlingOptions;

    /**
     * The path to the directory containing project config files (`package.json` or `tsconfig.json`)
     *
     * @default - the directory containing the `depsLockFilePath`
     */
    readonly projectRoot?: string;

    /**
     * The stack ID of Lambda@Edge function.
     *
     * @default - `edge-lambda-stack-${region}`
     * @stability stable
     */
    readonly stackId?: string;
}

export class NodejsEdgeFunction extends cloudfront.experimental.EdgeFunction {

    constructor(scope: core.Construct, id: string, props: NodejsEdgeFunctionProps) {
        const handler = props.handler ?? 'handler';
        const runtime = props.runtime ?? lambda.Runtime.NODEJS_14_X;
        const entry = path.resolve(findEntry(id, props.entry));
        const architecture = props.architecture ?? lambda.Architecture.X86_64;
        const depsLockFilePath = findLockFile(props.depsLockFilePath);
        const projectRoot = props.projectRoot ?? path.dirname(depsLockFilePath);
        super(scope, id, {
            ...props,
            runtime,
            stackId: props.stackId,
            code: bundling.Bundling.bundle({
                ...props.bundling ?? {},
                architecture,
                runtime,
                depsLockFilePath,
                entry,
                projectRoot,
            }),
            handler: `index.${handler}`,
        });
    }
}

/**
 * Checks given lock file or searches for a lock file
 */
function findLockFile(depsLockFilePath?: string): string {
    if (depsLockFilePath) {
        if (!fs.existsSync(depsLockFilePath)) {
            throw new Error(`Lock file at ${depsLockFilePath} doesn't exist`);
        }

        if (!fs.statSync(depsLockFilePath).isFile()) {
            throw new Error('`depsLockFilePath` should point to a file');
        }

        return path.resolve(depsLockFilePath);
    }

    const lockFiles = findUpMultiple([
        PackageManager.PNPM.lockFile,
        PackageManager.YARN.lockFile,
        PackageManager.NPM.lockFile,
    ]);

    if (lockFiles.length === 0) {
        throw new Error('Cannot find a package lock file (`pnpm-lock.yaml`, `yarn.lock` or `package-lock.json`). Please specify it with `depsFileLockPath`.');
    }
    if (lockFiles.length > 1) {
        throw new Error(`Multiple package lock files found: ${lockFiles.join(', ')}. Please specify the desired one with \`depsFileLockPath\`.`);
    }

    return lockFiles[0];
}

/**
 * Searches for an entry file. Preference order is the following:
 * 1. Given entry file
 * 2. A .ts file named as the defining file with id as suffix (defining-file.id.ts)
 * 3. A .js file name as the defining file with id as suffix (defining-file.id.js)
 * 4. A .mjs file name as the defining file with id as suffix (defining-file.id.mjs)
 */
function findEntry(id: string, entry?: string): string {
    if (entry) {
        if (!/\.(jsx?|tsx?|mjs)$/.test(entry)) {
            throw new Error('Only JavaScript or TypeScript entry files are supported.');
        }
        if (!fs.existsSync(entry)) {
            throw new Error(`Cannot find entry file at ${entry}`);
        }
        return entry;
    }

    const definingFile = findDefiningFile();
    const extname = path.extname(definingFile);

    const tsHandlerFile = definingFile.replace(new RegExp(`${extname}$`), `.${id}.ts`);
    if (fs.existsSync(tsHandlerFile)) {
        return tsHandlerFile;
    }

    const jsHandlerFile = definingFile.replace(new RegExp(`${extname}$`), `.${id}.js`);
    if (fs.existsSync(jsHandlerFile)) {
        return jsHandlerFile;
    }

    const mjsHandlerFile = definingFile.replace(new RegExp(`${extname}$`), `.${id}.mjs`);
    if (fs.existsSync(mjsHandlerFile)) {
        return mjsHandlerFile;
    }

    throw new Error(`Cannot find handler file ${tsHandlerFile}, ${jsHandlerFile} or ${mjsHandlerFile}`);
}

/**
* Finds the name of the file where the `NodejsFunction` is defined
*/
function findDefiningFile(): string {
    let definingIndex;
    const sites = callsites();
    for (const [index, site] of sites.entries()) {
        if (site.getFunctionName() === 'NodejsEdgeFunction') {
            // The next site is the site where the NodejsFunction was created
            definingIndex = index + 1;
            break;
        }
    }

    if (!definingIndex || !sites[definingIndex]) {
        throw new Error('Cannot find defining file.');
    }

    return sites[definingIndex].getFileName();
}

As we can see, I have taken many private functions from the NodejsFunction. The construct is now imported into my CDK stack as follows:

import { NodejsEdgeFunction } from './nodejs-edge-function';

...

const imageAccessFunction = new NodejsEdgeFunction(this, 'edge');

The Lambda@Edge code now behaves as we are used to from the NodejsFunction. The TypeScript Lambda can now be created in the src folder with the name STACK.edge.ts. Third-party libraries can be kept in the root package.json. Webpack as module bundler provides an efficient transformation into the JavaScript format.

Summary

Mega cool or? With a few steps, I can now write my Lambda@Edge in TypeScript as usual. Of course, I plan to contribute the code to the aws-cdk repo. Hopefully my PR will be accepted soon and you won't have to bother importing my code here. Do you find my construct helpful? If so write me!

Thanks to the DeepL translater (free version) for helping with translating to English and saving me tons of time :).

Become a Patreon!

CDK BitBucket Staging Pipeline Learnings (Part 2)

Fri, 07 Jan 2022 00:00:00 GMT

Hi CDK folks.

A few months ago I told you about my exciting project to build a CDK BitBucket Staging Pipeline. Since then, a lot has happened and we have continued to develop the pipeline.

Problems with CDK cross-stack references

CDK cross-stack references are CDK outputs passed as input to another CDK stack. Here is an example:

/**
 * Stack that defines the bucket
 */
class Producer extends cdk.Stack {
  public readonly myBucket: s3.Bucket;

  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const bucket = new s3.Bucket(this, 'MyBucket', {
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });
    this.myBucket = bucket;
  }
}

interface ConsumerProps extends cdk.StackProps {
  userBucket: s3.IBucket;
}

/**
 * Stack that consumes the bucket
 */
class Consumer extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props: ConsumerProps) {
    super(scope, id, props);

    const user = new iam.User(this, 'MyUser');
    props.userBucket.grantReadWrite(user);
  }
}

const producer = new Producer(app, 'ProducerStack');
new Consumer(app, 'ConsumerStack', { userBucket: producer.myBucket });

The example is taken from https://docs.aws.amazon.com/cdk/api/v1/docs/aws-s3-readme.html#sharing-buckets-between-stacks . You can see very well how the myBucket variable is created in the producer stack and how the consumer stack accesses it. This is a CDK cross-stack reference. And exactly those became problematic in our project.

If num changes something in the producer stack that would result in deleting and then recreating the myBucket variable, Cloudformation would respond with an error and cause a rollback. The reason is that since the consumer uses the cross-stack reference, it cannot be easily deleted. Such and other problems have made development difficult for us.

But we think and hope that we have found a good solution. First, we have reduced the number of stacks from about 7 to 4. The reallocation of services into the 4 stacks is now based on DDD (Domain Driven Design). This means that all services that belong to a domain, such as the React app, are bundled into one stack. Before, the division was rather random and based on the AWS services such as the LambdaStack or the CognitoStack. Now the stacks are assigned according to their domains and are called FrontendSiteStack, FrontendBackendStack and MLStack.

This new division has greatly reduced the number of cross-stack references. So that virtually only a few are left which we have outsourced to the CommonStack. However, the CommonStack serves as the parent stack to the other three. If now only cross-stack references between parent and children stacks exist, this should cause much less problems than references between sibling stacks.

Independent Stacks

The next challenge was that the developers would like to be able to deploy the three new stacks FrontendSiteStack, FrontendBackendStack and MLStack independently. Until now, all three were always deployed at the same time.

To solve this problem, we decided to use Custom Variables. Custom Variables are variables that can be set on Custom Pipelines during execution. To be able to deploy the stacks independently of each other, the custom pipeline must be executed with the following variables:

custom:
  cdkDeploy:
    - variables:
        - name: deployFrontendSite
          default: false
        - name: deployFrontendBackend
          default: false
        - name: deployML
          default: false

If you start the cdkDeploy pipeline with the BitBucket UI you can change the three variables to true if needed. A simple bash command then checks if the variable is true or not:

script:
  - if [ "$deployFrontendSite" != true ]; then exit 0; fi

So interactively by executing the custom pipeline and setting the variables the desired stack can be deployed. Very cool!

Stage

The concept of a stage can make the CDK code much cleaner. A stage groups all CDK stacks that belong to a stage like dev or prod. Here is an example.

export class Stage {

  /**
     * A stage which is a collection of stacks.
     *
     */
  constructor(scope: core.Construct, props: StageStackProps) {

    const commonStack = new CommonStack(scope, props.stage + '-CommonStack', { stage: props.stage, env: props.env });

    new FrontendBackendStack(scope, props.stage + '-FrontendBackendStack', {
      stage: props.stage,
      domainName: commonStack.domainName,
      zone: commonStack.zone,
      vpc: commonStack.vpc,
      env: props.env,
      userPoolId: props.userPoolId,
    });

    new MLStack(scope, props.stage + '-MLStack', {
      stage: props.stage,
      domainName: commonStack.domainName,
      zone: commonStack.zone,
      vpc: commonStack.vpc,
      env: props.env,
      enableAlarms: props.enableLambdaAlarms,
    });
  }
}

The stage is now simply imported into the main.ts:

new Stage(app, { stage: 'dev', env: devEnv, userPoolId: 'us-west-2_3zgoE123' });

By using a stage wrapper class, we have made our code much cleaner and easier to maintain.

What next?

A staging pipeline is already a good start to effectively and quickly develop new features and bring them into production. However, to develop even faster and more independent from other development teams, ephermal deployments are needed.

Ephermal Deployments means that with each new branch potentially a completely new CDK Deployment can be played out which contains the changes of the branch. So the developers don't have to share the dev environment anymore to test new features. I am curious to see how we will do this.

Our React frontend requires backend information at build time that is stored in a config file typical of AWS Amplify. These have to be laboriously kept up to date by hand. Therefore, it would be cool if we could enable the config file to be built dynamically before the build, similar to the Amplify CLI. Our initial idea is to enable this with AWS Api Gateway. The endpoint would then always send us the current backend data and we can save it as a config file.

Summary

I still enjoy working with BitBucket Pipeline. It can be used to build great CDK deployment pipelines. In this post I described what new challenges we had and how we solved them. I'm also very happy that the customer is now able to use the CDK pipeline themselves and is doing so more and more.

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

To the wonderful readers of this article I'm saying that feedback of any kind is welcome. In the future I will try to include a discussion and comment feature here. In the meantime, please feel free to send me feedback via my social media accounts such as Twitter or FaceBook. Thank you very much :).

Become a Patreon!

Asynchronous communication with AWS CDK EventBridge

Sun, 14 Nov 2021 00:00:00 GMT

Hi CDK folks

For a startup I am building a security platform. The rough process for using the platform is something like this. The customer answers a few questions and then gets a security evaluation. For the prototype I decided to use AWS EventBridge. With this I achieve a decoupled and asynchronous communication between the services.

This was the first time I used EventBridge and I am very excited about it. Since there isn't too much AWS CDK EventBridge material on the web yet, I wanted to share my experience with you.

Below I describe the Security Platform in a little more detail and what my architecture looks like.

Security Platform

The user can answer some questions using desktop app or mobile app. The answers are stored in the user service and sent asynchronously to the security service. Once the processing is done in the security service, the user gets the result presented as a report.

Architecture

The prototype has three modules. These three modules are each implemented with their own CDK stack. These are the frontend, the user service and the security service.

Frontend

The frontend is a ReactTS app hosted in S3 via static web hosting. The app uses Amplify libraries for user management like sign up, recover password, login, logout. Also I use Amplify's GraphQl libraries as typed access to the AppSync in the user service. Also feel free to check out my other posts about Appsync.

After the user loges in he will answer som security questions. The answers then are send and stored to the user service via GraphQL queries. The frontend also has a list of security reports created by the security service.

User service

Via AppSync the answers are stored in a DynamoDB table. From there, a DynamoDB Stream Lambda listens for new or changed answer records. The Lambda then writes the answer record as a AnswerEvent to the EventBridge EventBus. The EventBus forwards the AnswerEvent to the Security.

After the security service has processed the AnswerEvent, it sends a ReportEvent containing a report back to the user service via the EventBus. A Report Lambda in the user service then receives the ReportEvent and writes the report to a Report DynamoDB table via GraphQL mutation.

The frontend has a GraphQl subscription for newly appearing reports and thus the list of reports is automatically updated if a new report arrives.

Security Service

The security service receives an AnswerEvent from the user service. A Report Lambda then receives the AnswerEvent unwrappes the answer and processes it into a report. The report is then packed into a ReportEvent and sent back to the user service.

At the moment it is sufficient to use a Lambda to create the report. However, this could change in the future as the report will become more complex and, for example, will also use third party APIs. Then a more complex workflow could be required. With the use of this event architecture this should not be a problem.

EventBridge

To send events between the user service and the security service, I use the default EventBus. The AnswerEvent and ReportEvent look like this:

import { Answer, Report } from '../frontend/src/lib/api';

export interface AnswerEvent {
  answer: Answer;
  state: State;
}

export interface ReportEvent {
  report: Report;
}

Interestingly here is that for the definition of the events I simply reuse the GraphQL wrapper classes generated by AppSync in the frontend under ../frontend/src/lib/api.

Now I need to define the lambdas which pack the respective event into the default EventBus. Here for example you see the lambda which packs the AnswerEvent into the EventBus.

import * as AWS from 'aws-sdk';
const EventBridge = new AWS.EventBridge();

export async function handler(event: lambda.DynamoDBStreamEvent) {

  let answer: Answer | null = null;

  // unmarshall DynamoDBStreamEvent
  ...

  const answerEvent: AnswerEvent = {
    answer,
    state: State.new,
  };

  const params: AWS.EventBridge.Types.PutEventsRequest = {
    Entries: [
      {
        Detail: JSON.stringify(answerEvent),
        DetailType: DetailType.AnswerEvent,
        EventBusName: 'default',
        Source: Source.userServiceDdbStream,
        Time: new Date,
      },
    ],
  };
  const result = await EventBridge.putEvents(params).promise();
}

In the EventStack, rules and targets have to be defined.

const answerRule = new events.Rule(this, 'answerRule', {
  eventPattern: {
    source: [Source.userServiceDdbStream],
  },
});
answerRule.addTarget(new targets.LambdaFunction(props.ratingLambda));

const reportRule = new events.Rule(this, 'reportRule', {
  eventPattern: {
    source: [Source.ratingServiceLambda],
  },
});
reportRule.addTarget(new targets.LambdaFunction(props.getReportEventLambda));

Summary

Having services communicate with each other asynchronously is awesome and exciting. With AWS EventBridge and AWS CDK, it's also a wonderful developer experience. I am very excited to see where the security platform is going :). Let me know your thoughts!

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

A CDK BitBucket Staging Pipeline

Sun, 31 Oct 2021 00:00:00 GMT

I have been working for a client with very exciting AWS CDK tasks. The customer is heavily into the Atlassian ecosystem. BitBucket is used to host the code. Now the customer wants to move more into the DevOps space and manage their AWS deployments with AWS CDK as well. For this purpose, the existing AWS infrastructure needs to be translated into CDK. Additionally, a staging deployment pipeline should deploy the CDK stacks on a dev, qa and prod stages. Happy to help there :).

Disclaimer

Even though I already have a lot of experience with CDK (see here), I don't know much about BitBucket. So I don't know if my approach is ideal, but at the time of writing this article, it works quite well :).

I'm writing this article mainly because I haven't found any other helpful posts or instructions how to build a CDK staging deployment pipeline using BitBucket. So maybe if you have a similar task you might can use this post to help you getting started.

Which pipeline?

Now, of course, the question was where should the CDK staging deployment pipeline live? The choices were AWS CodePipeline or BitBucket's pipeline.

Maybe AWS CodePipeline?

The advantage of AWS CodePipeline would be that there is already an ingenious AWS CDK Staging Pipeline Construct like the CDK Pipeline . This would give you almost everything your DevOps heart desires, e.g. a synth action where the Cloudformation template is generated from CDK and deploy actions which then deploy to the respective stages. Also very useful are the optional actions that can be executed after the deploy action. Thus, for example, integration tests can be executed after the deploy. It is also very nice that the pipeline is defined in TypeScript. That provides documentation and specifies a certain standard through the typing for how a pipeline should look like.

Or BitBucket's pipeline?

The customer already uses BitBucket's pipeline for testing, linting and creating builds. Therefore, it would be undesired if the customer would be forced to switch back and forth between the two pipeline dashboards AWS CodePipeline and BitBucket's Pipeline. In addition, all functions of the AWS CodePipeline and the CDK Pipeline Construct can be reprogrammed with BitBucket's Pipeline.

A pretty cool feature that AWS CodePipeline doesn't have is skipping steps if the specified subdirectory hasn't been changed. Here is an example:

- step:
    name: Synth CDK app
    condition:
      changesets:
        includePaths:
          - "bitbucket-pipelines.yml"
          - "devops/**"
    script:
      - echo "synth cdk app"
      ...

I miss such a cool feature in AWS CodePipeline. Also, the customer uses a monorepo. So all projects are in one BitBucket repository. This would also be very disadvantageous for AWS CodePipeline since it would always trigger a pipeline run for every commit.

Considering all the pros and cons, we decided to use BitBuckets's Codepipeline as our CDK staging deployment pipeline.

Folder structure

As mentioned earlier, the customer has a monorepo and of course they would like to keep that. We have now decided on the following folder structure:

devops # Contains AWS CDK dependencies and CDK Apps
devops/${STAGE} # Contains stage specific scripts like bootstrap command
devops/${STAGE}/vpc # VPC CDK App
devops/${STAGE}/cognito # Cognito CDK App
devops/${STAGE}/website # S3 Website CDK App
...

STAGE is either dev, qa or prod .

In the devops folder are the AWS CDK dependencies. Via package.json all needed CDK libraries are loaded e.g. :

  "dependencies": {
    ...
    "@aws-cdk/aws-s3": "1.130.0",
    "@aws-cdk/aws-cloudfront": "1.130.0",
    "@aws-cdk/aws-cloudfront-origins": "1.130.0",
    "@aws-cdk/aws-s3-deployment": "1.130.0",
    ...
  }

Also, the devops/package.json includes a script to bootstrap the build AWS account and synthesize it. The build AWS account is the account from where the stages are managed. Synthesizing is the process of transforming the CDK app into one or more Cloudformation templates:

"scripts": {
  ...
  "synth": "yarn build && yarn cdk synth",
  "bootstrap": "yarn build && yarn cdk bootstrap --trust 11111111,222222,3333333 --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://44444444/us-east-1"
},

The bootstrap command creates a CDK helper stack in build account 44444444 and trusts all stages dev = 11111111, qa = 222222, prod = 3333333. Trusted here means that a cross account deploy permission is added to the CDK helper stack role. This allows the build account to deploy to the different stage accounts. It is also very important that the stage accounts must perform a bootstrap. The bootstrap command can be found in devops/${STAGE}/package.json e.g. for dev:

"scripts: {
  ...
  "bootstrap": "yarn cdk bootstrap --trust 44444444 --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://$111111/us-east-1"
}

As you can see here, the stage account e.g. dev=11111111 must also bootstrap the build account 44444444. These bootstraps can be a bit confusing, but fortunately don't need to be done often. I still recommend to document the bootstrap process well in e.g. the Readme.md . In general I think a good documentation is very very helpful :).

Example VPC

In each stage there should be a VPC where private infrastructure like Postgres DBs should be deployed. So each stage contains a VPC subdirectory devops/${STAGE}/vpc . There in a main.ts is the CDK code. I show this here with the dev stage:

import * as cdk from '@aws-cdk/core';
import { VpcStack } from '../../components/vpc-stack';
const env = require('../package.json').env;

export const DevVpcStack = (app: cdk.App) => new VpcStack(app, `${env.stage}-VpcStack`, { env });

Since each stage needs a VPC, it makes sense to place the common VPC CDK code in a shared component under the devops/components folder:

export class VpcStack extends cdk.Stack {

  vpc: ec2.Vpc;

  constructor(scope: cdk.Construct, id: string, props: VpcStackProps) {
    super(scope, id, props);

    this.vpc = new ec2.Vpc(this, 'vpc', { maxAzs: 2 });
  }
}

The respective VPC is then loaded in devops/src/main.ts:

import * as cdk from '@aws-cdk/core';
import { DevVpcStack } from '../dev/vpc/main';
import { ProdVpcStack } from '../prod/vpc/main';
import { SqaVpcStack } from '../sqa/vpc/main';
import { StagingVpcStack } from '../staging/vpc/main';

const app = new cdk.App();

// vpcs
const devVpc = DevVpcStack(app).vpc;
const sqaVpc = SqaVpcStack(app).vpc;
const stagingVpc = StagingVpcStack(app).vpc;
const prodVpc = ProdVpcStack(app).vpc;

...

Admittedly, artificially splitting the stages into devops/${STAGE}/vpc and then merging them into devops/src/main.ts is a bit weird and may not be ideal. Alternatively, everything could just be written to devops/src/main.ts. However, we hope to have a better overview of the "individual" CDK stacks this way.

Bitbucket pipeline

The Bitbucket pipeline now roughly goes through the following steps. First, tests are run in parallel and builds are build. Among the builds are, for example, different React builds for the different stages. After that, the CDK synth is run using yarn cdk synth. During synth, assets such as the different React builds are uploaded to S3:

- step:
  name: synth CDK app
  caches: 
    - node-custom
  condition:
    changesets:
      includePaths:
        - "bitbucket-pipelines.yml"
        - "devops/**"
  script:
    - export AWS_REGION=us-west-2
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_BUILD
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY_BUILD
    - cd devops
    - yarn synth
  artifacts:
    - devops/cdk.out/**

After the synth, the first stage dev can now apply the cloudformation templates synthesized by CDK. For this I use e.g. this CDK command:

yarn cdk deploy -a 'cdk.out/' dev-VpcStack --require-approval never

The cdk.out folder was previously designated as an artifact in the CDK synth step and is now reused for the respective stages (dev in this case) and CDK stacks. The pipeline yaml code looks like this:

- parallel:
    - step:
        name: Deploy vpc to dev
        caches: 
          - node-custom
        condition:
          changesets:
            includePaths:
              - "bitbucket-pipelines.yml"
              - "devops/**"
        script:
          - export AWS_REGION=us-west-2
          - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_BUILD
          - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY_BUILD
          - cd devops/dev/vpc && yarn deploy-cdk-stage
          - cd ../hasura && yarn deploy-cdk-stage
    - step:
        name: Diff qa
        caches: 
          - node-custom
        condition:
          changesets:
            includePaths:
              - "bitbucket-pipelines.yml"
              - "devops/**"
        script:
          - export AWS_REGION=us-west-2
          - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_BUILD
          - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY_BUILD
          - cd devops/qa && yarn diff
    - step:
        name: Approval deploy qa
        trigger: manual
        condition:
          changesets:
            includePaths:
              - "bitbucket-pipelines.yml"
              - "devops/**"
              - dashboard/**
        script:
          - echo "Deploy"
  - parallel:
        # qa stage simila looking to dev

So it is deployed directly to the dev stage. A diff to the next stage qa is also created in parallel. In order to also perform the qa deploy, the Approval deploy qa step must be approved via a manual trigger. If you agree, the whole process in qa runs analog to dev.

What next?

For the sake of clarity, we are still missing a kind of dashboard to display the most important CfnOutput URLs, such as the cloudfront urls of the React apps. We also want to know which commit was used for the respective deploy. For this purpose, a dashboard that can find out the current deployments and their commit ID via Lambda and probably also display them directly would be very suitable.

Summary

Working with BitBucket as a repository is mega cool. I had a lot of fun building a CDK staging deployment pipeline under these circumstances. Here I described roughly how this pipeline looks like and what our reasons were to build this pipeline this way. Only time will tell if these were good decisions when the customer actually uses the pipeline. Do you think we missed something or do you have ideas what can be done better? Then please write me :).

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

Security, Best Practice and Hardening Checks with the Prowler AMI

Fri, 27 Aug 2021 00:00:00 GMT

(Click me :D)

Hi Folks!

Developing in AWS is fun and the possibilities to build cool stuff seem endless. But are these cool things secure and do they follow AWS's best practices? With Prowler AMI you can check just that with far over 180 checks for many services in all your regions. For just $1, an Ec2 instance started with the Prowler AMI builds the cdk-prowler into your account. This will automatically start a CodeBuild that runs the checks. At the end you get an evaluation of the checks. The report contains information if the check was successful, the severity, a risk description and how to fix the problem.

In the next section I explain how exactly Prowler performs the checks.

Prowler Scan

The Prowler AMI uses the Github Prowler Tool. If you want to try Prowler locally, use the instructions from the Readme. Prowler can then be run either as a script with ./prowler or as a Docker container. Basically, Prowler then runs through the checks using the AWS CLI from the check Folder.

After running the checks, an HTML report is saved to an S3 bucket named prowleraudit-stack-prowlerauditreportbucket (see image from above). The HTML report now lists all findings of the performed checks. Since the recently released Prowler version 2.5.0 these can be filtered.

Run Prowler with AMI

As mentioned before you can deploy Prowler with my Prowler AMI to your account. In this section I will explain you how to do that.

First of course you click on the Prowler AMI link and subscribe to the AMI. Subscribing takes a little while. After the subscription is finished click on "Continue to Configure". Select the latest Prowler version and the region in which you want to run the Prowler checks. Then click on "Continue to Launch". Now very important at "Choose Action" you have to select "Launch through Ec2". Now you will be redirected to Ec2 in the AWS Console.

Now select the pre-selected instance type which should be t2.micro. Then click on "Next: Configure Instance Details". For the IAM role you need an Ec2 role that has administrator privilege. If the role does not exist yet click on "Create new IAM role" and create an AWS service role for Ec2 with the AdministratorAccess Policy. The Prowler AMI needs this permission to deploy the CDK stack on your system.

Alternatively you can call Prowler directly from the Ec2 AWS Console. Just go to Ec2 --> Instances --> Launch Instances in the AWS Console and search for Prowler. The rest is the same as described in the previous section.

Now the Prowler Cloudformation stack is deployed and the Prowler check is started using CodeBuild. At the end the HTML report ends up in the S3 bucket prowleraudit-stack-prowlerauditreportbucket .

The Ec2 instance is only needed to deploy the Cloudformation stack and should be terminated after the stack is finished to save costs.

Rerun Prowler with AMI

To run the Prowler scan again, you only run the AMI again. The Prowler stack will be triggered to run the scan again.

Summary

Prowler is a great tool to check your AWS account. With further over 100 checks, it gives me useful feedback on settings that don't follow best practices. Thus, I achieve a higher level of security in my AWS accounts. With my Prowler AMI it is very easy to run the tool in the account. Feel free to try Prowler AMI and tell me about your experience :).

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

Automated AWS Security Feedback with Prowler and AWS CDK

Tue, 17 Aug 2021 00:00:00 GMT

Hi Folks!

Toni De La Fluente has developed the super cool AWS Security Tool Prowler. Prowler is a cli tool for performing and consulting AWS Security Assessment, Auditing, Hardening and Incident Response. With more than 180 checks, Prowler provides the most comprehensive security checks offering for AWS. If you want to know more about Prowler visit the Prowler Github page.

I use Prowler a lot and decided to write an AWS CDK Custom Construct for it. In the past I have already created some custom constructs like my favorite the staging pipeline or the build badge.

However, if you just want to run Prowler for your AWS account and don't want to try the Custom Construct, I have good news. For the AWS Marketplace, I've created an AMI that automatically installs the cdk-prowler construct into your account. The AMI is available in the AMI Store. Just search for Prowler or use the Prowler AMI Link.

Then Prowler will run the security check and you will find the security findings in an S3 bucket named prowleraudit-stack-prowlerauditreportbucket the HTML report:

Or also in the codebuild report group:

If you have any problems with the AMI, please write me.

By buying the AMI you also support my free work to create such freely available products like the cdk-prowler custom construct and the work on my blogpost. Thank you very much :)!

In the next paragraphs I will explain why I think a CDK custom construct is useful and introduce you to the cdk-prowler custom construct. But first I have to explain what AWS CDK is.

AWS CDK

AWS CDK is an open source framework for creating and managing AWS resources. By using languages familiar to the developer such as TypeScript or Python, the Infrastructure as Code is described. In doing so, CDK synthesizes the code into AWS Cloudformation Templates and can optionally deploy them right away.

AWS CDK has been experiencing a steady growth of enthusiastic developers since 2019 and already has a strong and helpful community that is eg. very active on Slack. There is of course much more to say about AWS CDK and I recommend you explore it. Drop me a line if you have any questions.

Motivation

AWS CDK Custom Constructs are quasi own library to encapsulate cloudformation resources. They can be downloaded from registries like npmjs.com or pypi.org. So it is extremely easy to integrate cdk-prowler into your CDK IDE environment. Also it is very easy to make changes to the cdk-prowler code and provide a new version to the respective restries. Much of this important but tedious work is already greatly simplified by Projen AwsCdkConstructLibrary.

Another reason for the Custom Construct is the Jsii Library. With this mega cool library I can translate the cdk-prowler construct to different languages like Python, Java or CSharp and the developer can continue developing in his usual CDK language with cdk-prowler.

Beispiel

The cdk-Prowler Construct is on GitHub. It must first be included in the package.json as a dependency and can then be used in the CDK stack.


import { ProwlerAudit } from 'cdk-prowler';
...

const app = new App();

const stack = new Stack(app, 'ProwlerAudit-stack');

new ProwlerAudit(stack, 'ProwlerAudit');

app.synth();

The AWS CDK code shows how easy cdk-prowler can be deployed. Many additional functions like a scheduler are already available and can be activated and changed via the properties. Please have a look at the API.md.

Outlook

Many cool features for cdk-prowler are planned. The development of new features is done in cooperation with Tony De La Fluente the creator of Prowler.

Planned is to make Prowler multiaccount ready. So you have a centralized place for Prowler to check all accounts. Multiaccount should work with and without AWS organization.

In the future, Prowler will no longer be run with Codebuild but with Fargate. This allows a more flexible further development of Prowler to parallelize scans, for example. Parallelizing Prowler scans is an important and much needed feature because with over 180 Prowler tests running sequentially, execution can take a long time.

Also, Tony and I are thinking about an improved UI analysis tool with QuickSight. The Prowler HTML report is very useful but it is hard to send complex queries to the report. With QuickSight you have the possibility to define filters, variables, etc.

Summary

Prowler is mega cool and I love that it gives me automated feedback on security, best practices, etc. AWS CDK makes it easy to install ProwlerAudit and nest it with other services. Test the ProwlerAudit Construct and give me suggestions to make it even cooler. Feel free to create PRs for it in Github.

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

Wordpress Posts automated with AWS CDK

Sun, 08 Aug 2021 00:00:00 GMT

Hi Folks!

My last post was about how to make a wordpress deployment more DevOps friendly using Docker and some cool scripts. Today I want to talk about another exciting task. The customer wants new wordpress posts to be created automatically using AWS and when a video gets uploaded to S3. That sounds exciting so lets go. In the next few sections, I'll describe my solution.

Wordpress AWS solution

As described above, I want a wordpress post to be created after the video upload to S3. As well the video should be embedded in that post. For this I mainly need the two AWS services S3 and Lambda. The Lambda then needs to be set to run Lambda logic when uploading to S3.

The Lambda should then log into the wordpress environment via SSH and create a new post via WP CLI. I think it's pretty cool that wordpress has its own CLI. I can then use this to easily create new posts, for example.

The configuration of the AWS services I do with AWS CDK. You can find the complete code in my Github repo here

AWS CDK

The wordpress S3 Lambda stack now looks like this:

import { S3EventSource } from '@aws-cdk/aws-lambda-event-sources';
import * as lambdajs from '@aws-cdk/aws-lambda-nodejs';
import * as s3 from '@aws-cdk/aws-s3';
import * as cdk from '@aws-cdk/core';
import * as statement from 'cdk-iam-floyd';

export interface UploadBucketStackProps extends cdk.StackProps {
  readonly stage: string;
}

export class UploadBucketStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props: UploadBucketStackProps) {
    super(scope, id, props);

    const bucket = new s3.Bucket(this, 'my-videos', {
      bucketName: 'my-videos',
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
    });

    const lambda = new lambdajs.NodejsFunction(this, 'upload-trigger', {
      bundling: {
        externalModules: [
          'ssh2',
        ],
        nodeModules: [
          'ssh2',
        ],
      },
    });

    lambda.addToRolePolicy(new statement.Secretsmanager().allow().toGetSecretValue());

    lambda.addEventSource(new S3EventSource(bucket, {
      events: [s3.EventType.OBJECT_CREATED],
    }));
  }
}

For the Lambda I decided to use the NodeJsFunction Construct because it comes with many advantages like esbuild as package manager and I don't have to create a separate package.json for the Lambda because esbuild picks the needed library code from the main package.json.

The ssh2 module for the lambda has to be loaded as an external module though. This simply means that esbuild is not used as package manager for this and the module is loaded completely.

The code for the lambda function is shown and explained in the next section.

Lambda

The Lambda has as event source the S3 bucket and is executed when objects are created in the S3 bucket. Then the Lambda connects to the wordpress hoster via SSH and creates a wordpress post using WP CLI.

import * as lambda from 'aws-lambda';
import * as AWS from 'aws-sdk';
import { NodeSSH } from 'node-ssh';

var secretsmanager = new AWS.SecretsManager();

export async function handler(event: lambda.S3CreateEvent) {
  console.debug(`event: ${JSON.stringify(event)}`);

  const record = event.Records[0];

  const sshKey = await (await secretsmanager.getSecretValue({ SecretId: 'sshkey' }).promise()).SecretString;

  const ssh = new NodeSSH();
  await ssh.connect({
    host: 'mywordpresshoster.com',
    username: 'wp',
    privateKey: sshKey,
  });

  const objectKey = event.Records[0].s3.object.key;
  const videoLink = `https://${record.s3.bucket.name}.s3.${record.awsRegion}.amazonaws.com/${objectKey}`;

  // eg. 001_41_Das-ist-die-erste-Lektion.mp4
  // Kategorien: 41 - Grundlagen

  const fileName = objectKey.split('.')[0]; // eg. 001_kategorie1_Das-ist-die-erste-Lektion
  const videoFormat = objectKey.split('.')[1]; // eg. mp4
  videoFormat;
  const splittedFilename = fileName.split('_');
  const lection = Number(splittedFilename[0]); // eg. 1
  const category = Number(splittedFilename[1]); // eg. 41
  category;
  const title = splittedFilename[2].replace(/-/g, ' '); // eg. Das ist die erste Lektion


  console.debug(`video info: ${lection} ${category} ${title} ${videoFormat}`);

  const wpContent = `[vc_row][vc_column][vc_column_text]

Lektion ${lection}: ${title}

[/vc_column_text][us_separator][vc_column_text]

Here will be some content ...

<video poster="PATH-TO-STILL-IMAGE" controls="controls" controlsList=”nodownload” width="640" height="360">
    <source src="${videoLink}" type="video/mp4">
</video>

`;

  const wpCommand = `wp post create --post_title='${title}' --post_categories='${category}' --post_content='${wpContent}'`; // category?

  await ssh.execCommand(wpCommand/*, { cwd:'/var/www' }*/).then(result => {
    console.log('STDOUT: ' + result.stdout);
    console.log('STDERR: ' + result.stderr);
  });
};

As you can see I use the NodeSSH class from the node-ssh package to connect to the wordpress hoster. Then I pare some information from the S3 object key like the title or the lesson. At the end I just create the wordpress posts with the WP CLI.

Summary

Automating the creation of wordpress posts using AWS services is just mega cool. I built a small AWS CDK stack for this purpose. I'm already very excited about what else can possibly be automated in wordpress :). Do you have any ideas? Then write me!

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

Wordpress with Docker

Sat, 03 Jul 2021 00:00:00 GMT

Hi guys!

For my newest customer I built a great AWS backend with AWS AppSync and many other great AWS services. But now I'm faced with the exciting task of making the AppSync or GraphQL API callable in a Wordpress frontend. I have heard a lot about wordpress but never worked with it.

Also, this wordpress is hosted on Raidbox.de which makes it a bit difficult to manage the deployment directly. Coming from the DevOps world I want to configure the Wordpress deployment as comfortable and automated as possible.

I wanted three requirements to be met by the Wordpress deployment. First, Wordpress should be executable locally with minimal effort. Second, the Wordpress should be versioned with GitHub. And third, there should be a synchronization between the local wordpress and the wordpress hosted on Raidbox.

Wordpress

Wordpress is a content management system (CMS). It was developed to create blogposts. But it has developed a lot and is not only used for blogposts anymore. About so-called plugins new features are added to Wordpress. Such features can be a payment function. We ourselves use Digimember to manage a subscription payment system.

Wordpress with Docker

To solve my first problem, i.e. getting Wordpress to work locally, I decided to use Docker. Fortunately I was not the first one who tried or succeeded to do this. There are many good tutorials on the internet on how to set up a wordpress deployment with Docker. To orchestrate the Docker containers I use Docker Compose and here is the code:

version: '3.3'

services:
  wordpress:
    depends_on:
      - db
    image: wordpress:5.7.2
    volumes:
      - ./wordpress_files:/var/www/html
    ports:
      - "8080:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: wordpressuser
      WORDPRESS_DB_NAME: wordpress
      WORDPRESS_DB_PASSWORD: geheim
      WORDPRESS_HOME: 'http://localhost:8080'
      WORDPRESS_SITEURL: 'http://localhost:8080'

  db:
    image: mysql:5.7
    container_name: db
    volumes:
      - ./db_data:/var/lib/mysql
      - ./wordpress_files/wp-content/mysql-dump/wordpress.sql:/docker-entrypoint-initdb.d/1_wordpress.sql:ro
      - ./scripts/wordpress_replacing.sql:/docker-entrypoint-initdb.d/2_wordpress_replacing.sql:ro
    restart: always
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: geheim
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpressuser
      MYSQL_PASSWORD: geheim

  phpmyadmin:
    image: phpmyadmin/phpmyadmin
    container_name: phpmyadmin
    restart: always
    ports:
      - "8081:80"
    environment:
      PMA_HOST: db
      MYSQL_ROOT_PASSWORD: geheim

The first Docker container wordpress is of course for wordpress. With the volume wordpress_files I persist the wordpress files.

After that comes the db container which deploys a MySQL database. Wordpress uses MySQL as database to store information like blogposts, sites, users, pluginconfigs and much more. The volume db persists the database. The next two volumes store SQL scripts in the container at /docker-entrypoint-initdb.d :

- ./wordpress_files/wp-content/mysql-dump/wordpress.sql:/docker-entrypoint-initdb.d/1_wordpress.sql:ro
- ./scripts/wordpress_replacing.sql:/docker-entrypoint-initdb.d/2_wordpress_replacing.sql:ro

This will execute the SQL scripts on first startup. The first script wordpress.sql contains the complete Wordpress MySQL database which was previously exported as a dump. The second script does a text replacing from the domain to localhost:

UPDATE wp_options SET option_value = replace(option_value, 'https://www.example.com', 'http://localhost:8080') WHERE option_name = 'home' OR option_name = 'siteurl';
UPDATE wp_posts SET post_content = replace(post_content, 'https://www.example.com', 'http://localhost:8080');
UPDATE wp_postmeta SET meta_value = replace(meta_value,'https://www.example.com','http://localhost:8080');

This is necessary because I want to get the deployment working locally.

The last container phpmyadmin is a graphical administration for the MySQL DB.

GitHub

Pushing the code, the Wordpress files and the Docker Compose file to a GitHub repository is easy. Just create a new GitHub repository and copy and push the files to it.

Sync to Raidbox.de

As mentioned before, Raidbox.de is our hoster for Wordpress. To update the MySQL database and the Wordpress files there I decided to use rsync in combination with ssh. For this I have developed a workflow:

./scripts/syncWithStaging.sh pushes your local wordpress_files/wp-content, then dumps the staging mydb and then pulls the wordpress/wp-content including the dump. If there are changes now that probably means someone changed something directly over raidbox. Inspect the changes! If they are ok git commit the current status!
run docker-compose down && sudo rm -rf db_data && docker-compose up -d --build and make your changes
run ./scripts/createMySqlDumpExport.sh to export the local DB to mysql-dump-export
apply the mysql-dump-export/wordpress.sql to the wordpress with using raidbox phpmyadmin
run ./scripts/syncWithStaging.sh for syncing / pushing the wp-content folder.

This sync workflow may be changed in the future and the number of steps reduced. As it is designed now, you can still add e.g. blogs via the normal wordpress editor accessible via Raidbox. These new blogposts will then not be lost in the next sync workflow. In the following I show the used scripts

syncWithStaging.sh

#!/bin/bash

# push to remote
rsync -avzh --no-perms --no-owner --no-group -e "ssh -i ~/.ssh/example" ./wordpress_files/wp-content wp@b9emwoc.myraidbox.de:/home/wp/disk/wordpress

# create sql dump on raidbox.de
ssh -i ~/.ssh/example wp@b9emwoc.myraidbox.de "mkdir -p /home/wp/disk/wordpress/wp-content/mysql-dump && mysqldump -u wordpressuser --no-tablespaces --password=secret wordpress > /home/wp/disk/wordpress/wp-content/mysql-dump/wordpress.sql"

# pull from remote
rsync -avzh --no-perms --no-owner --no-group -e "ssh -i ~/.ssh/example" wp@b9emwoc.myraidbox.de:/home/wp/disk/wordpress/wp-content ./wordpress_files

Here you can see how with rsync, ssh and a private key (~/ssh/example) the wordpress files are first pushed and then pulled. Additionally the MySQL DB is dumped to Raidbox and pulled as well.

createMySqlDumpExport.sh

#!/bin/bash

# mysql export sql
CONTAINER=$(docker ps -aqf "name=db")
docker exec $CONTAINER /usr/bin/mysqldump -u wordpressuser --no-tablespaces --password=secret wordpress_staging > ./mysql-dump-export/wordpress.sql

# replacing http://localhost:8080 with https://www.example.com in the sql dump
sed -i s#http://localhost:8080#https://www.example.com#g ./mysql-dump-export/wordpress.sql

This script searches for the MySQL Docker container and creates a DB dump. Afterwards the urls are rewritten to the original url names. This dump can now be imported with the wordpress on Raidbox. Usually this is done with PHP Admin in Raidbox.

HTML as code

One thing that bothers me a lot with Wordpress is the inline editor for HTML and JavaScript code. Compared to e.g. Visual Studio Code it offers zero functionality. What I would like is that the HTML and JavaScript code sections in Wordpress can be done via local files.

I can then edit these local files as usual with my VS code editor and get great help like syntax correction, syntax highlighting and many more. I did this by converting the HTML and JavaScript into iFrames, which point to the files:

``html

```

But the whole thing probably still has one problem. It is not scalable or responsive when the size of the screen changes because you visit the wordpress site with a cell phone for example.

Summary

Wordpress is an interesting new experience for me. I like the wordpress editor which allows a graphical building of the website. However, I missed many out of the box DevOps techniques. But that doesn't matter with the approaches described here, development with Wordpress is much more fun.

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

AWS Cloudformation deploying with Lambda

Mon, 07 Jun 2021 00:00:00 GMT

Hi Folks!

I am currently working on an exciting project where it should be possible to create Ec2 instances flexibly via an API. These instances should then be made available to the customer for training purposes. The instances should be customized for the customer, so they should have a certain setting like the Ec2 type, some special tags and some more.

The deployment of the Ec2 instances should be done as cheap as possible by lambdas. About a year ago I did a similar project where Ec2 instances are dynamically created for an Alfresco deployment. Alfresco Provisioner

How to deploy Ec2 instances as dynamically as possible and also extremely easy to maintain with AWS CDK via an API like Api Gateway or Appsync, I will explain in the next sections.

AWS CDK

AWS CDK has been experiencing a steady increase in enthusiastic developers since 2019 and already has a strong and helpful community that is very active on Slack, for example. There is of course much more to say about AWS CDK and I recommend you explore it. Drop me a line if you have any questions.

In the next section, I briefly summarize the requirements for dynamic Ec2 deploying via an API.

Requirements

The Ec2 instances should be dynamically deployable via an API like AWS ApiGateway, which implements a REST API or AWS AppSync, which implements a GraphQL. In doing so, certain tags such as UserId or VmType shall be attached to the Ec2 instance. This allows the Ec2 instances to be easily retrieved and displayed in a React frontend, for example.

For the administration, i.e. creation, deletion and update of the resources belonging to the Ec2 instance, Cloudformation (short: CFN) should be used. Cloudformation is a handy AWS service for managing resources. Thus, the Ec2 stack can be easily created, modified or deleted. Another advantage for using CFN is that this part can be tested separately from the rest like the API for the Ec2 instance creation. I even created my own staging pipeline for this Stating Lib.

For the creation of the CFN templates I want to use AWS CDK, because using CDK makes the creation and maintenance of the Ec2 stack extremely easy.

The Ec2 CFN template synthesized by CDK is to be deployed through a Lambda.

Problem

Unfortunately, AWS CDK alone is not made for this use case. The main problem is that the CFN parameters cannot be flexible and after the synth phase the parameters were fixed. This drawback could be worked around by always doing a complete deploy with context parameters e.g.

cdk deploy -c userId=Alice -c vmType=2

does. But this is very impractical because it takes a lot of build time and every time a new CFN template is created.

To solve this problem I decided to use Cloudformation directly. Because CFN has the nice feature of parameters. So with every CFN deployment you can specify parameters to e.g. change the name of the Ec2 instance. Exactly what I was looking for.

Solution

So I use CDK for the Ec2 stack and the Lambda which should deploy the Ec2 stack dynamically. Let's start with the Lambda:

const cdkSchedulerLambda = new lambdajs.NodejsFunction(this, 'scheduler', {
  entry: `${path.join(__dirname)}/lambda/scheduler.ts`,
  bundling: {
    commandHooks: {
      afterBundling(inputDir: string, outputDir: string): string[] {
        return [`cp ${inputDir}/cfn/ec2-vm-stack.template.json ${outputDir} 2>/dev/null`];
      },
      beforeInstall(_inputDir: string, _outputDir: string): string[] {
        return [];
      },
      beforeBundling(_inputDir: string, _outputDir: string): string[] {
        return [];
      },
    },
  },
  logRetention: logs.RetentionDays.ONE_DAY,
  environment: {},
  timeout: core.Duration.minutes(15),
});

I use the LambdaJS variant here. This has the advantage that I can define the Lambda scheduler.ts directly in TypeScript and Construct will convert it to JS and upload it to AWS. That's as a great level of abstraction. And with the afterBundling webhook, I can upload the CFN template to Lambda already. So I don't have to do complicated work with another storage like S3 or EFS.

Next let's have a look at the scheduler.ts:

// eslint-disable-next-line import/no-extraneous-dependencies
import { readFileSync } from 'fs';
import * as lambda from 'aws-lambda';

import * as AWS from 'aws-sdk';
import { Ec2 } from './query-ec2';

const cfn = new AWS.CloudFormation();

export async function handler(event: lambda.DynamoDBStreamEvent) {
  console.debug(`event: ${JSON.stringify(event)}`);

  if (event.Records.length !== 1) {
    console.debug('event not valid! Exactly one record allowed!');
    return 'failed';
  }

  let newImage: Ec2 | null = null;
  if (event.Records[0].dynamodb?.NewImage) {
    newImage = AWS.DynamoDB.Converter.unmarshall(event.Records[0].dynamodb.NewImage) as Ec2;
  } else {
    console.debug('no NewImage existing');
  }

  let oldImage: Ec2 | null = null;
  if (event.Records[0].dynamodb?.OldImage) {
    oldImage = AWS.DynamoDB.Converter.unmarshall(event.Records[0].dynamodb.OldImage) as Ec2;
  } else {
    console.debug('no OldImage existing');
  }

  const templateBody = readFileSync('./ec2-vm-stack.template.json', 'utf-8');
  console.debug(`templateBody: ${JSON.stringify(templateBody)}`);

  if (newImage) {
    console.debug('Having NewImage so creating or updating');
    const createStackParams: AWS.CloudFormation.Types.CreateStackInput = {
      StackName: `stack-${newImage.userId ?? 'noUserId'}-${newImage.vmType ?? '-1'}`,
      TemplateBody: templateBody,
      Capabilities: ['CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM'],
      Parameters: [{
        ParameterKey: 'userIdParam',
        ParameterValue: newImage.userId,
      }, {
        ParameterKey: 'vmTypeParam',
        ParameterValue: newImage.vmType.toString(),
      }],
    };
    console.debug(`createStackParams: ${JSON.stringify(createStackParams)}`);
    try {
      const createStackResult = await cfn.createStack(createStackParams).promise();
      console.debug(`createStackResult: ${JSON.stringify(createStackResult)}`);
    } catch (error) {
      console.debug(`Creating failed with this error: ${JSON.stringify(error)}`);
      const updateStackResult = await cfn.updateStack(createStackParams).promise();
      console.debug(`updateStackResult: ${JSON.stringify(updateStackResult)}`);
    }
    return 'success';
  } else {
    if (oldImage) {
      console.debug('Having no NewImage but OldImage so deleting!');
      const deleteStackParams: AWS.CloudFormation.Types.DeleteStackInput = {
        StackName: `stack-${oldImage.userId ?? 'noUserId'}-${oldImage.vmType ?? 'noVmType'}`,
      };
      console.debug(`deleteStackParams: ${JSON.stringify(deleteStackParams)}`);
      const deleteStackResult = await cfn.deleteStack(deleteStackParams).promise();
      console.debug(`deleteStackResult: ${JSON.stringify(deleteStackResult)}`);
      return 'deleted';
    }
  }
  return 'failed';
};

By using a DynamoDBStreamEvent as input, it complicates the code a bit. In principle, however, these lines are important:

const templateBody = readFileSync('./ec2-vm-stack.template.json', 'utf-8');

const createStackParams: AWS.CloudFormation.Types.CreateStackInput = {
      StackName: `stack-${newImage.userId ?? 'noUserId'}-${newImage.vmType ?? '-1'}`,
      TemplateBody: templateBody,
      Capabilities: ['CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM'],
      Parameters: [{
        ParameterKey: 'userIdParam',
        ParameterValue: newImage.userId,
      }, {
        ParameterKey: 'vmTypeParam',
        ParameterValue: newImage.vmType.toString(),
      }],
    };
    console.debug(`createStackParams: ${JSON.stringify(createStackParams)}`);
    try {
      const createStackResult = await cfn.createStack(createStackParams).promise();
      console.debug(`createStackResult: ${JSON.stringify(createStackResult)}`);

      ...

So here the CFN template is loaded from the ec2-vm-stack.template.json file and applied using AWS SDK createStack.

So all that remains is the ec2-vm-stack.template.json. How do I create this? Well it is as already said also a CDK stack. However, this one uses CFN parameters as input for userIdParam and vmTypeParam. And here is the code for the Ec2 stack:

import * as ec2 from '@aws-cdk/aws-ec2';
import * as cdk from '@aws-cdk/core';
import { CustomStack } from 'aws-cdk-staging-pipeline/lib/custom-stack';
import { KeyPair } from 'cdk-ec2-key-pair';
import * as statement from 'cdk-iam-floyd';

export interface Ec2StackProps extends cdk.StackProps {
  readonly stage: string;
}
export class Ec2Stack extends CustomStack {
  constructor(scope: cdk.Construct, id: string, props: Ec2StackProps) {
    super(scope, id, props);

    const vpc = ec2.Vpc.fromLookup(this, 'VPC', {
      isDefault: true,
    });

    const securityGroup = new ec2.SecurityGroup(this, 'sg', {
      vpc: vpc,
    });

    securityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(22));
    securityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(80));

    const userData = ec2.UserData.forLinux();
    userData.addCommands(`
#!/bin/bash
yum update -y
yum install -y httpd.x86_64
systemctl start httpd.service
systemctl enable httpd.service
echo “Hello World” > /var/www/html/index.html
    `);

    const userIdParam = new cdk.CfnParameter(this, 'userIdParam', {
      default: 'noUserId',
    });

    const vmTypeParam = new cdk.CfnParameter(this, 'vmTypeParam', {
      default: '-2',
    });

    const identifier = `${userIdParam.value.toString() ?? 'noUserId'}-${vmTypeParam.value.toString() ?? '-1'}`;

    const instance = new ec2.Instance(this, 'instance', {
      instanceName: `vm-${identifier}`,
      instanceType: new ec2.InstanceType('t2.micro'),
      vpc,
      securityGroup,
      keyName: 'mykey',
      machineImage: ec2.MachineImage.latestAmazonLinux({
        generation: ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
      }),
      userData,
    });

    cdk.Tags.of(instance).add('Owner', 'Hacklab');
    cdk.Tags.of(instance).add('UserId', userIdParam.value.toString());
    cdk.Tags.of(instance).add('VmType', vmTypeParam.value.toString());

    instance.addToRolePolicy(new statement.Ec2().allow().toDescribeVolumes().toDetachVolume()
      .toAttachVolume().toCreateTags().toDescribeTags().toTerminateInstances().toDeleteSecurityGroup().toDescribeInstances().toStopInstances());

  }
}

Interestingly, I use the CFN parameters userIdParam and vmTypeParam for the dynamic part of the Ec2 stack. If the stack is now created as via the Lambda, for example, the Ec2 instance gets the two tags UserId and VmType with the values of the CFN parameters.

Summary

Creating Ec2 instances using CFN and Lambdas is a cool thing. With CDK everything i.e. the lambda, the API and the Ec2 stack can be formulated and extended very easily. Super cool is also that with CFN it is very easy to add other resources in the future. For example, in the future I may want to provide an S3 bucket for deployment as well. I hope you enjoyed this article and maybe it even helped you. Do you have any questions? Write me :) !

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

AWS CDK Custom Construct Versions Checker

Tue, 18 May 2021 00:00:00 GMT

Hi CDK Fans!

With Custom Constructs it is possible to implement certain features as Level 2 AWS CDK Constructs. In fact, many of the AWS CDK Constructs already in use are already Level 2 Constructs. I have already built some custom constructs. There is the Build Badge and the Staging Pipeline.

As we all know AWS CDK iterates very fast and a new version is released basically weekly. With each release there is a risk that my custom constructs are not compatible with the released CDK version. Therefore, I have developed a GitHub workflow that automatically tests the compatibility of my custom constructs against the new version.

In the next sections I will explain how the compatibility check works and how I implemented it.

AWS CDK

AWS CDK is an open source framework for creating and managing AWS resources. Using languages familiar to the developer such as TypeScript or Python, the Infrastructure as Code is described. In doing so, CDK synthesizes the code into AWS Cloudformation Templates and can optionally deploy them right away.

In the next section I explain how the compatibility check works with the AWS CDK version.

Compatibility check

Once a day a GitHub workflow, here called version checker, curls to the page https://github.com/aws/aws-cdk and extracts the latest released CDK version. Then the version checker creates a branch in the name pattern bump/1.X.X . So when version 1.103.0 was released recently, the version checker created the branch bump/1.103.0 . Then the version checker creates a pull request against master. The pull request now contains the new CDK version. This is achieved by the version checker making a string replace in the .projenrc.js file. For example:

const cdkVersion = '1.97.0';

replaced with

const cdkVersion = '1.103.0';

Now a normal build is performed but with a new CDK version. The build and unit tests checks the compatibility with the CDK dependencies automatically. The use of the version checker requires a good unit testing strategy. Lambda code testing could look like that e.g.:

import * as AWS from '../__mocks__/aws-sdk';
import { handler } from '../src/index.badge';

const codebuild = new AWS.CodeBuild();

describe('test index.badge.ts lambda', () => {
  describe('failure when', () => {
    test('queryStringParameters.projectName is not existing', async () => {
      mockedApiEvent.queryStringParameters.projectName = undefined;
      await handler(mockedApiEvent).catch((reason: any) => {
        console.log(`reason: ${reason}`);
        expect(JSON.stringify(reason)).toContain('projectName in query parameter is not existing or empty!');
      });
    ...

and the expected synthesized cloudformation templates tests can look like this:

import { SynthUtils } from '@aws-cdk/assert';
import * as core from '@aws-cdk/core';
import { BuildBadge } from '../src';
import '@aws-cdk/assert/jest';


describe('Get', () => {
  describe('BuildBadge', () => {
    const app = new core.App();
    const stack = new core.Stack(app, 'testing-stack');

    describe('successful', () => {
      test('with not defined hideAccountID', () => {
        new BuildBadge(stack, 'BuildBadge1');
        expect(stack).toHaveResourceLike('AWS::ApiGateway::Method');
        expect(stack).toHaveResourceLike('AWS::Lambda::Function');
        expect(JSON.stringify(SynthUtils.toCloudFormation(stack))).toContain('\"ACCOUNT\":\"123\"');
      });
      test('with hideAccountID = "no"', () => {
        new BuildBadge(stack, 'BuildBadge2', { hideAccountID: 'no' });
        expect(stack).toHaveResourceLike('AWS::ApiGateway::Method');
        expect(stack).toHaveResourceLike('AWS::Lambda::Function');
        expect(JSON.stringify(SynthUtils.toCloudFormation(stack))).toContain('"ACCOUNT\":{\"Ref\":\"AWS::AccountId\"}');
      });
      ...

These examples were copied from my Build Badge Construct.

Versionschecker - GitHub Workflow

The version checker is a GitHub workflow and is run daily at 4 o'clock.

name: cdkversioncheck
on:
  schedule:
    - cron:  '0 4 * * *'
  # push:
  #   branches:
  #     - master

jobs:
  check:
    runs-on: ubuntu-latest
    env:
      CI: "false"
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      # - name: Set CDK version manually
      #   run: |-
      #     echo "CDK_VERSION=1.99.0" >> $GITHUB_ENV
      - name: Fetch CDK version
        run: |-
          echo "CDK_VERSION=$(curl --silent "https://api.github.com/repos/aws/aws-cdk/releases/latest" |
          grep '"tag_name": "v1' |
          sed -E 's/.*"v([^"]+)".*/\1/')" >> $GITHUB_ENV
      - name: Synthesize project files
        continue-on-error: true
        run: |-
          yarn add projen
          echo "${{ env.CDK_VERSION }}"
          sed -i "3s/.*/const cdkVersion = '${{ env.CDK_VERSION }}';/" .projenrc.js
          npx projen
      - name: Set git identity
        run: |-
          git config user.name "Auto-bump"
          git config user.email "github-actions@github.com"
      - name: Push branch
        run: |- 
          git checkout -b bump/${{ env.CDK_VERSION }}
          git diff --exit-code || ((git add package.json yarn.lock .projen/deps.json .projenrc.js) && (git commit -m "Testing CDK version to ${{ env.CDK_VERSION }}" && git push -u origin bump/${{ env.CDK_VERSION }}))
      - name: pull-request
        uses: repo-sync/pull-request@v2
        with:
          source_branch: bump/${{ env.CDK_VERSION }}
          pr_title: Testing CDK version to ${{ env.CDK_VERSION }}
          pr_label: "cdk-version-test"  
          destination_branch: "master"
          github_token: ${{ secrets.PUSHABLE_GITHUB_TOKEN }}
    container:
      image: jsii/superchain

The curl for the version number only accepts v1.X tags. v2.X is by design not compatible with my CDK stacks. So the PR would always fail.

It would also be possible to test manually against specific versions. For this some lines of code have to be commented out or in. E.g. for manual triggering the push master must be commented. Also the action to curl the CDK version must be commented out and the CDK version action above it must be commented in.

Tested Versions List

I had the cool idea to create a list of the already tested CDK versions. Simply the created pull requests are provided with the previously created label cdk-version-test.

- name: pull-request
  uses: repo-sync/pull-request@v2
  with:
    source_branch: bump/${{ env.CDK_VERSION }}
    pr_title: Testing CDK version to ${{ env.CDK_VERSION }}
    pr_label: "cdk-version-test"  
    destination_branch: "master"
    github_token: ${{ secrets.PUSHABLE_GITHUB_TOKEN }}

Then you can simply link to a filter with the label like https://github.com/mmuller88/aws-cdk-build-badge/pulls?q=is%3Apr+is%3Aopen+label%3Acdk-version-test . And voila all PRs for the version test are displayed.

Summary

AWS CDK is still the perfect framework for Infrastructure as Code in AWS. With my Custom Constructs I hope to give back to the community. To ensure the best experience at all times, I have developed a workflow that checks CDK version compatibility. Creating the workflow was a lot of fun. But even more fun is to have a PR list where I can see if the version checks worked. I have more exciting CDK topics in my forum: https://martinmueller.dev/tags/cdk

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

AWS QuickSight DataSet and DataSource CDK Custom Constructs

Thu, 06 May 2021 00:00:00 GMT

Hi.

In my previous blog posts I showed how to generate analytics from DynamoDB table using AWS QuickSight. Unfortunately, QuickSight is only minimally supported by Cloudformation and QuickSight's DataSource and DataSet are only on the roadmap so far https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/274 .

Therefore, I have developed AWS CDK Custom Constructs for DataSource and DataSet. In this post, I go into a little more detail about what QuickSight DataSources and DataSets are and how I deploy using AWS CDK Custom Construct.

AWS CDK

AWS CDK is an open source framework for creating and managing AWS resources. Using languages familiar to the developer such as TypeScript or Python, it describes the infrastructure as code. In doing so, CDK synthesizes the code into AWS Cloudformation Templates and can optionally deploy them right away.

DataSource

A QuickSight DataSource defines the source of data for analysis. You can see which data sources are possible in the SDK Documentation. The DataSource construct also uses the SDK parameter definition as typed input.

The concrete interface can be found here. For example if you want to define Athena as DataSource it looks like this:

const users=['martin.mueller'];

const datasource = new DataSource(this, 'DataSource', {
  name: 'cdkdatasource',
  dataSourceParameters: {
    athenaParameters: {
      workGroup: 'ddbworkgroup',
    },
  },
  users,
});

dataSourceParameters is the typed parameter from the SDK. I think it's pretty cool that we have type support here now and so we can also define other sources as DataSource like Aurora or RDS very easily <3 . The workGroup in the example must be created before. You can do that manually with the AWS Console in Athena or even better with Athena. I already explained how exactly to do this in a previous blogpost.

DataSet

A DataSet can then be used to refine and concretize the DataSources. For example joins or transforms can be defined. All possibilities you can find here in the DataSet SDK Documentation. Very important to know, the permissions for DataSource and DataSet must always be set correctly. I have already built this into the constructs for example.

The implementation of the DataSet is here. In the following I describe a more complex example where logical tables are joined from the DataSource.

new DataSetConstruct(stack, 'DataSetConstruct', {
  name: 'cdkdataset3',
  users: users,
  physicalTableMap: {
    users: {
      customSql: {
        dataSourceArn: datasource.dataSourceArn,
        name: 'users',
        sqlQuery: 'SELECT primarypractice, dateofbirth FROM "ddbconnector"."martin1"."martin1" WHERE groupid = \'users\' AND firstname is not null',
        columns: [
          { name: 'primarypractice', type: 'STRING' },
          { name: 'dateofbirth', type: 'STRING' },
        ],
      },
    },
    practices: {
      customSql: {
        dataSourceArn: datasource.dataSourceArn,
        name: 'practices',
        sqlQuery: 'SELECT id, name FROM "ddbconnector"."martin1"."martin1" WHERE groupid = \'medical-practices\' AND name is not null',
        columns: [
          { name: 'id', type: 'STRING' },
          { name: 'name', type: 'STRING' },
        ],
      },
    },
  },
  logicalTableMap: {
    'users': {
      alias: 'users',
      source: {
        physicalTableId: 'users',
      },
    },
    'practices': {
      alias: 'practices',
      source: {
        physicalTableId: 'practices',
      },
    },
    'users-practices': {
      alias: 'users-practices',
      source: {
        joinInstruction: {
          leftOperand: 'users',
          rightOperand: 'practices',
          type: 'INNER',
          onClause: 'primarypractice = id',
        },
      },
      dataTransforms: [{
        createColumnsOperation: {
          columns: [{
            columnName: 'age',
            columnId: 'age',
            expression: 'dateDiff(parseDate(dateofbirth, "YYYY-MM-dd\'T\'HH:mm:ssZ"),now(), "YYYY")',
          }],
        },
      }],
    },
  },
});

In this example, the two tables users and practices are first created virtually and then joined together to find out to which practice the user is assigned.

I also perform a data transformation here. I would like to know the age of the user, but the data only contains the date of birth. So I perform a transformation where the age is calculated and defined as a new column named age.

Summary

QuickSight is a cool AWS tool for visualizing data insights. Since I am a huge fan of Infrastructure as Code I want to have all QuickSight resources in CDK. Unfortunately, the DataSource and DataSet are not yet supported by Cloudformation. As long as this is still the case, I and you are welcome to use my CDK Custom Construct as a replacement :).

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

AWS DynamoDB Analysis with QuickSight and AWS CDK - Quite big Tables

Wed, 21 Apr 2021 00:00:00 GMT

Hi.

In my last blogpost, I covered exciting work with AWS Athena and AWS QuickSight. If you want to do analytics from small AWS DynamoDB tables, everything should work smoothly. By small I mean a relatively small number of columns in the table.

It happened to me that when processing the TAKE2 data, the required columns for QuickSight were not displayed at all. Now the number of columns in the TAKE2 data is anything but small with more than 700! How I solved the problem and even cast it into code using AWS CDK, you will learn in the next sections.

Before that I would like to thank the sponsor TAKE2 for this blogpost as well.

Solution

The problem is that the DynamoDB table is way too big and the Lambda AthenaDynamoDBConnector is no longer reasonably able to recognize all the columns. Fortunately, the programmers of the connector have considered this case and have built in a possibility to define specific columns with their names and types.

The exact instructions are in this repo. In short, an AWS Glue Table must be created. The connector can then detect the desired columns based on the columns defined there.

To visualize it using the AWS component diagram also used in the last post, all you need to do is add a Glue Table:

AWS CDK Code

I packed the CDK Glue Table extension into its own CDK Stack. Here is the code snipped:

import * as glue from '@aws-cdk/aws-glue';
import * as cdk from '@aws-cdk/core';

interface GlueStackProps extends cdk.StackProps {
  readonly ddbTableName: string;
}

export class GlueStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props: GlueStackProps) {
    super(scope, id, props);

    const database = new glue.Database(this, 'Database', {
      databaseName: props.ddbTableName,
      locationUri: 'dynamo-db-flag',
    });

    const gluetable = new glue.Table(this, 'GlueTable', {
      tableName: props.ddbTableName,
      database: database,
      columns: [{
        name: 'userid',
        type: glue.Schema.BIG_INT,
      }, {
        name: 'firstname',
        type: glue.Schema.STRING,
      }
      ...
      ],
      dataFormat: glue.DataFormat.JSON,
    });

    const cfngluetable = gluetable.node.defaultChild as glue.CfnTable;
    cfngluetable.addPropertyOverride('TableInput.Parameters.classification', 'dynamodb');
    cfngluetable.addPropertyOverride('TableInput.Parameters.columnMapping', 'userid=userId,firstname=firstName,...');
  }
}

The DynamoDB columns you would like to be parsed you simply need to define as columns in the glue table. Watch out here! Glue Table does not support certain characters like capslock and special characters. Therefore you may have to do a columnMapping!

After deployment the columns defined in columns can be used in Athena and QuickSight.

Summary

Analyzing large DynamoDB tables is not that easy. However, with the use of AWS Glue Tables, the Lambda AthenaDynamoDBConnector can still reliably detect the desired columns.

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

AWS DynamoDB Analysis with QuickSight and AWS CDK

Thu, 08 Apr 2021 00:00:00 GMT

Hi.

AWS DynamoDB is an extremely performant and scalable NoSQL database. Due to the lack of a schema, data, called items in DynamoDb, can be extremely flexible. This also allows a kind of evolutionary development of the items by simply creating new columns.

But there is a catch. Because we are no longer in the world of relational data, we can no longer perform relational operations such as joins or the usual relational operations like COUNT, ORDER BY, GROUP by and many more. Now you ask yourself why should I be able to perform joins for example, they already bugged us with relational databases? We can use joins for example in analysis. I'll try to explain it with a store:

Top X products sold in the period from t1 to t2 grouped by gender.

In our example the users and the sold products are each in their own DynamoDB table and are indirectly connected via a userId. However, it is also conceivable that both are in the same table, but in different data sets or rows. DynamoDB does not allow joins and so we cannot relate products sold to gender.

The solution to the problem is AWS Athena, QuickSight, Lambda and S3. Using a Lambda, we store the DynamoDB items as a flat JSON file in an S3. Then we let Athena access it. QuickSight then uses Athena as a data engine to create joins, analytics, and dashboards. How you automate this with AWS CDK and a description of the AWS services used will follow in the next sections. For the impatient ones, here is the code.

But before we got to the next section I would like to thank the sponsor for this blogpost and the exciting project to perform analysis of DynamoDB tables using QuickSight. Thanks to TAKE2 for letting me be part of your agile and motivated team to work on exciting AWS CDK tasks like these.

AWS DynamoDB

AWS DynamoDB is a managed NoSQL database with very good performance and scaling. Managing the database from AWS eliminates tedious administrative tasks such as installation or maintenance. DynamoDB also has backup features like on demand or point-in-time recovery.

In DynamoDB, the inserted data does not have to follow a fixed schema like in relational databases. This is super flexible and very useful, but can also lead to problems like confusion or inconsistencies in column names. Therefore I recommend to allow only certain columns in the table. This can be achieved for example by schema validation in Api Gateway or using a GraphQL schema in AWS AppSync.

AWS Athena

AWS Athena allows data to be queried. To access the data, the developer uses standard SQL as the query language. The data source can be various AWS services such as S3, RedShift and most recently DynamoDB. The advantage to Athena is that it is serverless, so you can focus directly on querying the data.

To set DynamoDB as data source for Athena you need a Lambda Connector. The connector writes all items from the table into an S3 bucket. Fortunately, AWS already provides a SAM Lambda that does the job. This lambda is called AthenaDynamoDBConnector

AWS QuickSight

AWS QuickSight is a service for creating and analyzing visualizations of customer data. The customer data can reside in AWS services like S3, RedShift or as in our case in DynamoDB.

QuickSight cannot directly read data from DynamoDB at this time and a small intermediate step must be taken. The DynamoDB data has to be exported in a S3 bucket e.g. as JSON. Then QuickSight can read the data located in the S3.

To push the data into the S3 bucket the approach of using an AthenaDynamoDBConnector Lambda is suitable. You can read more about this in the next section.

Quicksight offers many cool functions for processing and visualizing data that can come from DynamoDB for example. You can also define them as code with CDK.

I am working on storing the QuickSight analyses in templates via CDK in order to make them cross account accessible. This will allow me to create and test analyses on a dev account and then deploy them automatically to the prod account via CDK. How exactly this should work will be explained in the next blogpost.

AWS CDK

AWS CDK](https://github.com/aws/aws-cdk) is an open source framework for creating and managing AWS resources. By using languages familiar to the developer such as TypeScript or Python, the infrastructure is described as code. In doing so, CDK synthesizes the code into AWS Cloudformation Templates and can optionally deploy them right away.

AWS CDK has been experiencing a steady increase in enthusiastic developers since 2019 and already has a strong and helpful community that is very active on Slack. There is of course much more to say about AWS CDK and I recommend you explore it. Drop me a line if you have any questions.

With AWS CDK, I have achieved a high level of automation in creating and managing the DynamoDB QuickSight deployment. This involves defining the required AWS resources and its configurations nicely as code and then simply executing them. And here you can see a component diagram:

You can find the AWS CDK code for the DynamoDB Athena deployment in my repo. Be sure to check out the readme there as well since it contains a lot of important instructions and information. Unfortunately I couldn't write everything in AWS CDK because e.g. the SAM Lambda AthenaDynamoDBConnector is not or only difficult to translate in AWS CDK and it has to be deployed manually until now.

Outlook

It would be super cool if the AthenaDynamoDBConnector was also available in AWS CDK. Also, the QuickSight Cloudformation resources still seem very immature to me as some things are not supported at all like the DataSet. A GitHub Issue has already been created.

As mentioned in the QuickSight section, I am working on persisting the analytics from QuickSight using CDK and then making them available cross account wise. I will show my findings in the next blogpost.

Summary.

AWS QuickSight is an exciting analytics tool for evaluating data in a DynamoDB table. Data is the new gold and therefore it is extremely important to be able to process data that is difficult to process, for example in a DynamoDB table. If you can use familiar mechanisms like SQL queries and aggregate functions, that's great. Also QuickSight offers cool graphical solutions for displaying analyses and dashboards. I am very excited to be able to work with it more.

Thanks also to Jared Donboch for the extremely helpful BlogPost Using Athena data connectors to visualize DynamoDB data with AWS QuickSight . Based on that, I was able to automate as much as possible by composing it into an AWS CDK stack.

Thanks again to TAKE2 for sponsoring this blog post.

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

A very cool DevOps AWS CDK Todolist

Tue, 23 Feb 2021 00:00:00 GMT

Hi.

For a long time I've been dreaming of building a platform with a great DevOps experience. I want to have cool stuff in it like:

Infrastructure as Code (IAC)
CI / CD (Continuous Integration / Continuous Deployment)
A staging e.g. dev, qa, prod
automated tests
all this with as little TypeScript code as necessary

I have found that now with the use of the following technologies. AWS CDK to manage the infrastructure as code. AWS AppSync as implementation of GraphQL, AWS Amplify for the frontend website and managing users. Furthermore AWS CodePipeline extended with my library AWS CDK Staging Pipeline.

With my Todolist deployment I want to show you how much power is in my deployment. But first I will explain what AWS CDK is.

AWS CDK

AWS CDK is an open source framework for creating and managing AWS resources. By using languages familiar to the developer such as TypeScript or Python, Infrastructure as Code is described. In doing so, CDK synthesizes the code into AWS Cloudformation Templates and can optionally deploy them right away.

AWS CDK has been experiencing a steady increase in usage for developers since 2019 and already has a strong and helpful community that is very active like on Slack. There is of course much more to say about AWS CDK and I recommend you explore it. Drop me a line if you have any questions.

Backend AWS AppSync, AWS DynamoDB and AWS Cognito

The code for the backend is in GitHub aws-cdk-todolist.

AWS AppSync is a GraphQL api implementation. GraphQL is an interesting new API technology. Since GraphQL is still new for me, I don't dare to make any big statements now. What I do like is that the connection to databases like AWS DynamoDB feels very native. I've always been annoyed by the necessary Lambda implementations with AWS Api Gateway and the creation of new endpoints. All of that seems no longer necessary with the use of AWS AppSync.

For storing items in the todo list I use DynamoDB. Each item gets its own ID, which is assigned by the AppSync resolver. In addition, each item has a body and a username.

AWS Cognito is a service for user management and user authentication via Oauth. This service makes it very easy to implement a professional and scalable identity service with all the bells and whistles.

A deployment pipeline deployed all these resources to AWS accounts. The AWS resources for the backend and the deployment pipeline itself, are deployed using AWS CDK as Infrastructure as Code.

Front-end React, AWS Amplify, S3 Static Website.

The code for the frontend is in GitHub aws-cdk-todolist-ui.

The frontend stack consists of a React build with AWS Amplify UI extensions like @aws-amplify/auth @aws-amplify/ui-components @aws-amplify/ui-react. These extensions help massively with user management. For example, it is easy to create new users via the login screen or change the password in case you forget it.

The React build is copied to an S3 bucket via the deployment pipeline (see next section). The S3 bucket is configured as a static website. All required AWS resources for the S3 bucket and the deployment pipeline are also provided with AWS CDK.

DevOps Pipelines

Any deployments today should have a decent deployment pipeline that has cool features like automated builds, deploys, tests, linting, versioning and so on. With this you achieve an incredible speed in development and the work is much more fun because annoying manual tasks like deploying to production or manual testing are simply omitted.

My Todolisten deployment has such a deployment pipeline. Each of the frontend and backend have their own pipeline which is based on AWS CodePipeline and of course AWS CDK. For the pipeline I have developed a custom AWS CDK construct and also reported about it in a previous post which you can see here.

Outlook

The AppSync Transformer looks very interesting. With it it is possible to use the AWS Amplify GraphQL Transformer more easily. Thus many lines of code can be saved by generating an extended GraphQL schema using cool labels like @model @connection.

I do make a Udemy course about creating a platform like the Todolist. If you want a voucher for it, let me know and once it's done you'll get one.

Summary

GraphQL is ultra exciting. I think it's a forward looking API technology and I can't wait to work with it more in a professional environment.

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

Scrum and GitHub

Thu, 11 Feb 2021 00:00:00 GMT

Hi.

Our development team does an exciting experiment. We work in Scrum and organize ourselves mainly with GitHub. That works really great1. In this article I describe how we use GitHub for that.

Even though I have experience with Scrum for about 5 years as a developer I give a little disclaimer here. I have never done a Scrum certification. So I will probably misrepresent or misclassify terms. I would then like to ask you to correct me.

Also, our team has only been working together in Scrum for a little over a month. The team is relatively small with 4 people 3 fullstack developers and 1 product owner. Nevertheless I think this post can help you to improve your work process in your team.

In the next section I'll tell you more about GitHub and how we use it to organize ourselves in Scrum.

GitHub and Scrum

My GitHub account is active since 2016. GitHub is by far my favorite Git provider environment. Privately, I use GitHub a lot. I am very happy that I can now use GitHub for work as well. Our main motivations to use GitHub are the low costs and the fact that many developers are already used to GitHub.

In the next section I explain how we use some GitHub features to manage our Scrum way of working.

GitHub Issues as Tickets

The issues on GitHub can be perfectly used as tickets for Scrum. Using labels it is even possible to give the tickets a size. A size in this context is a number to estimate the complexity of the ticket. Usually these are the Fibonacci numbers 1 3 5 8 13 .

Labels can also be used to assign tickets to their respective components. Component means a kind of subcategory of the project like "component: api" or "component: app" .

GitHub Projects as Scrum Boards

GitHub Projects are great for Scrum Boards. We have 3 Projects for 3 Boards. Each board represents its own time range and level of abstraction.

Current Sprint Board

The most important board is of course the "Current Sprint Board" (see in the middle of the cover image). We do a two-week sprint and with the "Current Sprint Board" we reflect on the work or the tickets we are currently working on and want to work on. It has three columns ToDo, In Progress and Done.

Tickets from the Backlog Scrum Board, which will be introduced in the next section, end up here first in the ToDo column. At this point, the tickets should be understandable and sized. At the end of each Sprint, the Done column is emptied and the ToDo column is refilled.

Backlog Board

The Backlog Board (see below in the title image) is intended for tasks in the period of 2 - 6 weeks and includes all tickets that are not yet in the current Sprint Board. Here they are collected and refined piece by piece. There are four columns from left to right: Backlog, Ready for Sizing, Ready for Sprint and Next Sprint.

In each column, the ticket is refined, e.g. when the ticket is moved from Backlog to Ready for Sizing, it must be comprehensible to all developers so that it can be sized.

When the ticket goes into the Ready for Spring column, it is now ready to potentially go into the next sprint. In the Next Sprint column we agree on which tickets we want to handle in the next sprint.

Future Board

The Future Board is for ideas / tasks that are still relatively unclear and are in a time interval of about 6 weeks to a year. So far we hardly use this board and have ideas in a separated Miro board.

Summary

I am still totally flashed how well it works with the tools on GitHub to work in Scrum. The UI of GitHub seems very sophisticated and allows a great Scrum experience. Just give it a try :) !

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

Docker Hub release with Projen

Mon, 01 Feb 2021 00:00:00 GMT

Hi Projen folks.

Projen is becoming popular. I've been using it myself for about half a year now and I'm still totally hooked. Recently I even managed to build a Docker Image Release Cycle to Docker Hub with it. My Docker InfluxDB Repo now automatically pushes an image to Docker Hub on every release.

But what is Projen and how I use it? That and more I'll explain in the next sections.

Projen

Projen allows sophisticated management of project configuration through code. With just a few lines of TypeScript, an entire repository can be configured. Here is an example:

const { JsiiProject } = require('projen');

const project = new JsiiProject({
  name: 'alps-unified-ts',
  authorAddress: 'damadden88@googlemail.com',
  authorName: 'Martin Mueller'
});

project.synth();

These few lines are creating all the GitHub project files you need. Among them the package.json, .gitignore, tsconfig.json and many more.

What's cool is that Projen also comes with GitHub workflows that can for example publish new versions to registries like NPM or PYPI which I used for an other projen which I describe here. For my Docker image release to Docker Hub I extended the Projen GitHub release workflow. You can see how exactly in the next section.

Projen Setup

Below you can see my current Projen setup which is stored in the file .projenrc.js.

const { NodeProject, ProjectType, DockerCompose } = require('projen');

const project = new NodeProject({
  authorAddress: 'damadden88@googlemail.de',
  authorName: 'Martin Mueller',
  name: 'influxdb-s3-backup',
  releaseBranches: 'main',
  projectType: ProjectType.LIB,
});

project.releaseWorkflow.addJobs({
  publish_docker_hub: {
    needs: 'build',
    'runs-on': 'ubuntu-latest',
    env: {
      CI: "true",
    },
    'steps': [
      {
        name: 'Check out the repo',
        uses: 'actions/checkout@v2',
      },
      {
        name: 'Set up QEMU',
        uses: 'docker/setup-qemu-action@v1',
      },
      {
        name: 'Set up Docker Buildx',
        uses: 'docker/setup-buildx-action@v1',
      },
      {
        name: 'Login to DockerHub',
        uses: 'docker/login-action@v1',
        with:{
          username: '${{ secrets.DOCKER_USERNAME }}',
          password: '${{ secrets.DOCKER_PASSWORD }}',
        }
      },
      {
        name: 'get_version',
        id: 'get_version',
        run: [
          'DVERSION=$(jq .version version.json -r)',
          'echo "::set-output name=dversion::$DVERSION"',
          ].join('\n'),
      },
      {
        name: 'Build and push',
        uses: 'docker/build-push-action@v2',
        with: {
          context: '.',
          file: './Dockerfile',
          platforms: 'linux/amd64,linux/arm64',
          push: true,
          tags: 'damadden88/influxdb-s3-backup:${{ steps.get_version.outputs.dversion }},damadden88/influxdb-s3-backup:latest'
        }
      },
    ],
  }
})

project.synth();

NodeProject

My project is from type NodeProject:

const { NodeProject, ProjectType, DockerCompose } = require('projen');

const project = new NodeProject({
  authorAddress: 'damadden88@googlemail.de',
  authorName: 'Martin Mueller',
  name: 'influxdb-s3-backup',
  releaseBranches: 'main',
  projectType: ProjectType.LIB,
});

This means that it is a typical node deployment, which requires nodejs as environment and contains additional libraries and project information in the package.json. Also, Node projects come with a changelog.md which can list all changes since the last release.

It also has a GitHub release workflow which automatically makes GitHub releases on commits using GitHub Actions and follows the semver. I have now extended this release workflow to also push Docker images to Docker Hub in the same tag version as it is done during the GitHub release. To extend the release workflow I have extended the releaseWorkflow object as follows:

project.releaseWorkflow.addJobs({
  publish_docker_hub: {
    needs: 'build',
    'runs-on': 'ubuntu-latest',
    env: {
      CI: "true",
    },
    'steps': [
      {
        name: 'Check out the repo',
        uses: 'actions/checkout@v2',
      },
      {
        name: 'Set up QEMU',
        uses: 'docker/setup-qemu-action@v1',
      },
      {
        name: 'Set up Docker Buildx',
        uses: 'docker/setup-buildx-action@v1',
      },
      {
        name: 'Login to DockerHub',
        uses: 'docker/login-action@v1',
        with:{
          username: '${{ secrets.DOCKER_USERNAME }}',
          password: '${{ secrets.DOCKER_PASSWORD }}',
        }
      },
      {
        name: 'get_version',
        id: 'get_version',
        run: [
          'DVERSION=$(jq .version version.json -r)',
          'echo "::set-output name=dversion::$DVERSION"',
          ].join('\n'),
      },
      {
        name: 'Build and push',
        uses: 'docker/build-push-action@v2',
        with: {
          context: '.',
          file: './Dockerfile',
          platforms: 'linux/amd64,linux/arm64',
          push: true,
          tags: 'damadden88/influxdb-s3-backup:${{ steps.get_version.outputs.dversion }},damadden88/influxdb-s3-backup:latest'
        }
      },
    ],
  }
})

So this GitHub workflow releases my image to Docker Hub with the same tag version and latest:

tags: 'damadden88/influxdb-s3-backup:${{ steps.get_version.outputs.dversion }},damadden88/influxdb-s3-backup:latest'

New releases to GitHub and Docker Hub are now completely managed by Projen automatically. This is extremely cool and saves me a lot of time :).

Summary

I love the Projen framework as it is a great abstraction and default for a setup of projects. I encourage you to give Projen a try and very soon you will experience the beauty of this framework yourself.

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

IOT - Gasmeter goes digital

Mon, 25 Jan 2021 00:00:00 GMT

Before it begins. Affiliates regarding the IOT devices are in the post

Hi.

A few weeks ago I presented my new IOT deployment here. It consists of a Raspberry 4 DE US UK, Zigbee Türsensor DE UK and the Zigbee Empfäger DE UK. The software that runs on the Raspberry 4 is Docker Compose. With the Docker Compose deployment, NodeRED is used for IOT management. For the charts which you can see in the picture I use Grafana.

First I want to explain how I made my analog gas meter digitally and then how I use NodeRED and Grafana to transform the received data and make it displayable in charts (see title image).

Gas meter extension

As shown in the picture above right I have an analog gas meter BK-G4. So how do I turn it into a digital gas meter that can even store values over time? Well the meter sends a magnetic pulse every 0.01 qm of consumed gas. On the picture you can see this as 1 imp 0,01 qm. So if you also have a gas meter with magnetic pulse, the instructions here should also work for you.

I simply measure the magnetic pulse with the wireless Zigbee door sensor (also top right in picture) and send it to a Zigbee receiver which is connected to a Raspberry 4 US and record the values.

The recording is not that easy because the signal has to be transformed. The transformation of the signal is done with NodeRED.

NodeRED

NodeRED is an open source programming tool for connecting IOT devices together, APIs and online services. It provides a web browser based editor and supports JavaScript as a programming language. The runtime itself is Nodejs.

I also find it very cool that NodeRED supports a connection to GitHub and the current state of the NodeRED project can be saved versioned using the Git protocol. NodeRED offers many other great features that you should definitely explore. But I only want to talk about the NodeRED Flow.

NodeRED Flow

A NodeRED project can have several flows. A NodeRED Flow describes the interaction of the IOT devices with other devices, APIs or online services. You can see my flow in the upper left corner of the image. The flow is best read from left to right. On the far left are two inputs. The lower input is from an Open Weather API that I use to collect the current outdoor temperature (also shown in yellow in the "Gas Delta" diagram).

The upper input is from my Zigbee receiver which gets the signal from the Zigbee door sensor. The Zigbee door sensor and receiver work with the so called MQTT protocol. MQTT is a data saving, publish and subscribe protocol mainly for communication between IOT devices.

On the right side of the flow you can see the outputs (a bit cut off). The blue output is a NodeRED UI element. NodeRED also offers the possibility to render UI elements with e.g. sensor values. This is then accessible e.g. at https://localhost:1880/ui . The brown outputs send data to an InfluxDB which is then displayed with Grafana.

InfluxDB & Grafana

InfluxDB is an open source time series database . It is written in GO and optimized for fast, high availability storage and query of stored data. It is great for IOT data.

For my flow, I store the current gas value and temperature in InfluxDB. InfluxDB then calculates the timestamp on its own. Then I can simply use the graphical analysis tool Grafana to get the data from InfluxDB and display it flexibly (see diagrams in title image).

At the bottom right I see the current and yesterday's daily consumption. You can not see it in the picture because only yellow is displayed but I have set different colors for color ranges. From 0 - 7 qm is green. Then from 7 - 14 qm is yellow and everything above 14 qm is red. For a family of three the average daily consumption is about 7 qm, which varies by a factor of 2 at most in summer and winter. Extremely cool ^^.

At the very bottom left, the total consumption of the gas is displayed and above it how the consumption changes relatively over the day. A pretty cool feature of Grafana is that the time range can be changed comfortably. As default I have set "Today" which shows me all values from 0 to now. But there are other time ranges like yesterday, the day before yesterday, 12 hrs etc. Also start and end time can be entered directly. Extremely cool!

InfluxDB AWS S3 Backup

In the meantime, I even added a data backup feature. With this I compress my InfluxDB data and store it in AWS S3. AWS S3 is a cheap long term cloud storage. I prefer this method because I don't have to buy external storage and can leave the physical management of the storage to AWS.

For backup, I built my own Docker image and release. This was necessary as there was no Docker Image on Docker Hub yet which supported InfluxDB + AWS CLI + arm64. I now consume these in my rasp4 repo in docker-compose.yaml. Backup is done by default every morning at 1am.

Outlook

Next I would like to digitallise my electricity meter to be able to read it as well and show nice charts.

I also want to migrate more IOT devices to NodeRED because I currently have some things running in the cloud with the app "Smart Life". But the functionality there is very limited. E.g. I can already turn my boiler on and off and I would like to be able to control that with NodeRED.

Summary

If you have any cool ideas how to transform or visualize my sensor data to generate more useful information, let me know.

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

Long term investing with Trading212

Sun, 17 Jan 2021 00:00:00 GMT

Before it starts! The Trading212 link is an affiliate where you and I get each a share for free.

Hi,

I started with stocks in November 2020 and since them I am an active user of the trading app Trading212. The main reason is that I do not see a bank account as an ideal form of storage for my money since it loses value there over time.

The exact explanation why I invest in stocks and how my strategy looks like, you will see in the next sections. Only so much is said, it was a great decision and I have already created after a few weeks a higher value than it would be possible in a bank in a year.

Why?

Banks nowadays only offer a very low interest rate up to a maximum of 2% per year. This can hardly or not at all keep up with the inflation rate of money. In other words, you lose financial strength. To better understand the word financial strength, I'll give a small example. Lets imagine in year 2020 you could afford 100 buns of 100€, in 2021 it only are 90 buns you can buy with 100€.

So if you leave your money in the bank, you lose financial strength. So other options should be considered when it comes to storing money. An investment in shares with a low risk strategy sounds very tempting at first. At this point, however, it should be said that every investment carries a risk, and this of course also applies to shares. Even banks can go bankrupt at the expense of investors, see Greece.

I also find the idea of supporting certain companies by buying shares very exciting. For companies, the stock market is a good opportunity to get interest-free capital to invest. Alternatively, the company would have to apply for an interest-bearing loan from a bank. To a certain extent, I can invest in companies that I find great. I like companies with the topics:

Green Energy: Companies that promote the expansion and development of renewable energies such as solar panels or wind turbines.
Quality Food: I am a strong supporter of high quality food and I like the topics like organic, regional, fair trade.
US Tech Companies: As a software developer, I naturally know a lot about tech companies and how useful and valuable they are to humanity.
Non-US Tech Companies: Being from Germany myself, I would specifically like to support tech companies in the EU area.

And I can actively decide against companies with certain topics like fossil fuels or alcohol as a fun drink. These were my personal topics and they can look completely different for you, which is also ok.

In the next section I explain how I invest in stocks with Trading212.

Trading212

Trading212 is an internet platform for investing in stocks. Trading212 advertises that it does not charge a commission, which makes the platform very attractive. If you use a broker, they usually charge a commission. Trading212 also have an iOS and Android app and of course a web interface.

I myself use the Trading212 ios app as well as the web interface and am very satisfied. It makes it very easy to invest in stocks and track their performance over the long term. Depositing and withdrawing from my bank account to my Trading212 account is also very easy and fast.

Last year (2020) a great feature called Pie was released. With this it is possible to create so called Pies in which you can create up to 50 shares. Each share can then be assigned to a certain percentage value. In the sum all percentage values result then naturally again 100%. If you want to know what else the Pies can do just click here.

I find the Pies great because they help with my low risk strategy and give me a great overview of how my broad portfolio is developing. I also like the AutoInvest feature where I can pay a certain amount into the Pie every month.

Quite recently, the "Share Pie" feature has been added. With this feature you can share the created pies and others can see how my pie looks like, i.e. in which shares and how many percent I have invested. Concrete amounts of money are not revealed. And this here is my Pie with a diverse choice of stocks and companies I want to support.

Even though the range of available stocks to invest in at Trading212 is very large, it happened once or twice that my desired stock was not available. In such a case, however, you can ask in the Trading212 Community Forum to include the respective share in the portfolio of Trading212.

The Trading212 Forum Community is the perfect place if you want to learn more about trading with Trading212 and in general. The community there has been very supportive in my investment strategy for which I am very grateful. Recently Trading212 has also added a Discord Channel.

Also great is that Trading212 offers free webinars on trading on their platform (so apps too) as well as on YouTube. In the next section I explain my investment strategy.

My strategy

I want to invest long term (more than 10 years) in many different sectors like IT, materials, energy etc. What sectors there are and what they mean you can find here. Investing money in many different sectors instead of one specific sector reduces the risk of loss because these sectors are more independent from each other than one sector would be. If you look at the Corona crisis, for example, you see that the IT sector did not suffer as much as real estate, for example.

Modern internet platforms like Trading212 allow to do without the middle man like the broker, who usually charges a commission, because well designed platforms make it very easy to take care of your portfolio yourself.

Furthermore, I would like to deposit a part of my monthly income automatically with AutoInvest into my Pie. This way I can build up a good reserve over the years without much effort. I start with 100€ which will be transferred automatically every month and increase this amount in the future if necessary.

Since I am still quite new in the subject of shares, I will invest about 20-30% of the portfolio into ETFs. And as my experience increases over time, swap out the ETFs for regular stocks. ETFs are managed by professional traders and this management of course costs something. My long term goal is to do without ETFs completely, because Trading212 with the Pies allows me to manage my own ETFs and I can save the costs.

Outlook

I am already very satisfied with my current selection of stocks and will probably only make minimal improvements. Then I would swap stocks and ETFs that are not performing so well for more promising ones. Let's see.

My investments are so far mainly concentrated on the American, German and British stock market and I should think about making my portfolio even more diverse and invest in other stock markets and how exactly I split into these.

I am very excited about the "Pies in Pie" feature that would simplify the management of my portfolio even more. Currently I still need an Excel spreadsheet to keep track of how much I have invested in which sectors. With Pies in Pie this would be directly readable.

Also an export feature should come soon, where I can export the investments e.g. into an Excel sheet. With this I can then feed other visualization tools to make interesting analyses. But I don't have any concrete ideas yet.

Summary

Trading212 makes investing in stocks fun. It gives me all the tools I need to manage my stock portfolio myself. It's a great alternative to holding money in accounts and you learn a lot of useful things about the stock market. I think it is important and right to develop your financial intelligence. Trading212 can be your start to do just that!

Thanks to the DeepL translater (free version) for helping with translating to english and saving me tons of time :).

Become a Patreon!

RC3 Summary

Tue, 05 Jan 2021 00:00:00 GMT

Hi friends of the chaos,

The 37th Congress has taken place from 27 - 30 December 2020 and due to the pandemic everything was remote. So the Chaos Congress got the name RC3. This was my very first Chaos Computer Club event that I attended. How great the days were and what I experienced, I describe in the next paragraphs.

Tag 1

Visited streams:

12:20 #rc Eröffnung
13:00 Die Geschichte der Corona-Warn-App
14:00 Flutter - Introduction
14:30 Wie gründe ich eine Genossenschaft?
16:00 Rage Against The Machine Learning
19:00 What the cyberoptimists got wrong-and what to do about it

It's December 27th, 2020, and RC3 starts at 12:20. The intro was very cool. There was a summary of the year 2020 and besides Corona many interesting and unfortunately also sad things happened to which attention was drawn. It was also shown and made clear how much effort and how many people are actually involved in making the RC3 possible. Extremely cool.

The first talk I attended was 1pm "The History of the Corona Warning App". It was really exciting to follow how the German Corona Warn app has evolved, as I had barely followed it. I found it very cool that the conference was remote and I was partly in 3 streams at the same time to pick the best one. That would not work in reality ^^

At 14:30 my highlight stream was about "How to start a cooperative?". The topic of starting a cooperative interests me a lot, as I'm in the process of starting one with a few other developers. So it was really cool to learn in detail what a cooperative is and to hear about the experiences of people who have already founded one. For me it was very interesting to hear that cooperatives can probably help to save taxes in some areas and that cooperatives are probably underestimated in the general public, although they are something very great.

I attended many other streams on the day and it's always great to see people so excited to present about their topics.

Also extremely cool and fun and awesome was RC3 World which is a digitized two-dimensional world of the RC3 fairgrounds and more. You create a little avatar there and walk around with many others in a 2 d world. You can also talk to the others and gaze at each other via webcam if you get close enough. Extremely funny and cool. Among other things, the Chaos Spaces of Germany were modeled in the world and the Hacklabor is very well done (see cover picture).

Tag 2

Visited streams:

11:00 Daten : IT bei der Polizei
12:20 Scientific Literacy 101
13:00 Mundhalle
14:20 Flutter - An advanced view on cross-platform app development
15:00 Hackergenossenschaft Plattform chaosnahen Wirtschaftens
19:00 Gesundheit ist eine soziale Frage

Today is Monday and the second day of rC3. My highlight stream today was the 15:00 "Hacker cooperative as a platform for chaos-related business" . Being in the process of starting a cooperative myself, it's highly exciting to hear about it. I found it a bit disturbing that the hacker cooperative presented there took a year to found the cooperative. I think that has a lot to do with the bureaucracy in Germany. However, I am optimistic that our cooperative will be founded more quickly.

Also very exciting was the 19:00 "Health is a social question" talks, which showed very nicely how the current German health system is ailing. E.g. that prevention plays a rather subordinate role. It also made me think about why patients are actually released back into their environment, which has made them so ill. This is exactly where one should actually start and address the environment / the living conditions of the patient.

Tag 3

Visited streams:

11:05 Digitales Ehrenamt meets Deutsche Stiftung fuer Engagement und Ehrenamt met
13:00 Offene Daten für Viele Nutzbar machen

Today I didn't have that much time to watch streams because I was working on projects. In the evening I did some socialising at the rC3 World. I met some cool dudes from the Rostock Space. I realized that I know some of them from my time as a student in Rostock. The rC3 World collapsed at some point and it took a long time until it was available again.

Day 4

Today I couldn't visit any streams because I had to do a lot at work. But I was able to spend some time with friends in the world and watch some streams.

Summary

Many thanks to the organizers and contributors of rC3. It was my very first Chaos Computer Club event and it just knocked my socks off. There were many exciting streams and the world was awesome. For the fact that this was the first rC3, the relatively frequent technical marbles / failures are definitely forgivable.

Anyhow I hope for 2021 the C3 will take place in person again, but if not the rC3 is a great alternative! Thanks also to my buddy and experienced C3 congress visitor for all the little tips and the fun get-togethers that made my rC3 experience great :).

I love to work on Content Management Open Source projects. A lot from my stuff you can already use on www.github.com/mmuller88 . If you like my work there and my blog posts, please consider supporting me on Patreon:

Become a Patreon!

Raspberry 4 IOT with AWS CDK Pipeline automated Deployment

Sat, 02 Jan 2021 00:00:00 GMT

Hi Raspberry Friends.

Note: I have used commission links in this article and marked them with "*".

Since recently I can call myself owner of a Raspberry. On Cyber Monday I bought a *Raspberry 4 with 4 GB. I have had previous experiences with Home IOT and summarized them in a blogpost. However, these experiences were tied to ready-made IOT solutions and therefore give only a limited feature set. A good friend of mine, who has a lot of experience with Home IOT (Internet of Things), showed me what more is possible.

So I aimed for the next IOT level with building cool solutions using MQTT and NodeRED. My first use case is that I want to read the gas meter digitally and get nice evaluation graphics displayed in relation to time. On the internet I found a tutorial how to read my gas meter, digitally supported BK-G4. But this solution required a complex NodeRED deployment with the use of a Raspberry.

The Raspberry can be treated like a normal VM. That means it is loaded with an operating system like Ubuntu. Then necessary programs are installed and afterwards the NodeRED Deployment is executed. If needed, the VM can simply be thrown away and everything run again. This is a lot of manual steps which can be very time consuming and error prone.

Therefore, I decided to automate the entire deployment to a high degree using the following technologies GitHub, Docker, Docker Compose, AWS CodeDeploy Agent, AWS CodePipeline with CDK Pipeline. Basically, then all you need to do is push to the main branch and the entire Docker deployment builds and launches itself.

I will describe how this all works in the next sections.

Setup

First, however, I would like to describe my setup. As mentioned before, I bought a Raspberry 4 (Rasp4 for short) with 4GB for Cyber-Monday *https://amzn.to/3rD3HOf. Furthermore there was a 64GB microSSD in the set.

The installation of Ubuntu 20.04.1 was very easy and is very well documented on ubuntu.com . I chose the Ubuntu server because I want to use the Rasp4 as a headless remote VM.

To read the gas meter I needed two more components. A *Zigbee door/window sensor and a Zigbee USB stick as receiver. The sensor sends a signal to the Zigbee receiver every time the meter measures 0.1 sq ft of gas consumption. This is a specific solution for my gas meter BK-G4. It may not be possible for your gas meter.

Installation of the environment

The IOT programs are to be run using Docker Compose and deployed from my GitHub Repo the Docker Compose YAML file. Furthermore, the Docker Compose deployment should be automatically rebuilt and redeployed as soon as something is pushed to the main branch.

To make all this possible many programs like Docker, Docker Compose, Git, AWS CLI, AWS CodeDeploy Agent etc, are needed. To simplify the installation of the required programs I created a script ./misc/init.sh. This is then simply executed on the Rasp4:

sudo chmod +x ./init.sh
./init.sh

Important to mention. The Rasp4 has only a one way internet connection. That means it has no public IP and is not reachable from outside. It is only reachable in the local network.

Automated Deployment with AWS CodeDeploy & AWS CodePipeline

To reduce manual effort as much as possible, I decided to automate the Docker Compose deployment. For this I use an AWS CodeDeploy agent that automatically registers changes on the main branch and executes them on the Rasp4. The installation of the AWS CodeDeploy agent was a bit tricky but worked out in the end. The agent is also installed with the init.sh.

To have the agent perform actions on the Rasp4 when commits happen on the main branch, an AWS CodePipeline must be created. The AWS CodePipeline is configured and managed with AWS CDK . AWS CDK is an open source framework for creating and managing AWS resources using high level languages like TypeScript or Python. The AWS CDK code can also be found in my GitHub repo and here.

If I made you curious about AWS CDK, you can find countless other blogposts about AWS CDK on my blog page.

IOT Docker Compose Stack

My IOT stack consists of the following containers NodeRED, InfluxDB, Grafana, Mosquitto and Zigbee. The exact settings are in the docker-compose file . I assembled the stack using the Docker Compose Builder from IOTstack.

I recommend you use one of the forks from IOTstack as the user gcgarner doesn't seem to maintain the builder anymore.

What did I enjoy the most?

Extremely cool was the easy installation of Ubuntu on my Rasp4. This even worked without an extra keyboard and monitor and is called remote installation. You just have to enter the wifi credentials during the image creation on the microSSD card. This simple process reminded me somehow of Docker images, which are also very light on their feet. Extremely cool!

It also happened a few times that I reloaded the entire OS to solve configuration issues. Since I had all the steps of the installation in init.sh, that was no problem.

Also, creating the AWS CodePipeline with AWS CDK was extremely cool. Since I already have a lot of experience with AWS CDK, it was very easy to do. Infrastructure as code is just great!

Difficulties

Setting up the AWS CodeDeploy agent was very time consuming and annoying, as it only works with certain program versions of awscli or ruby. To this day I'm not sure why the agent always bitched so much. Well it works now and that's what counts.

Outlook

I have already started with the NodeRED Flow and can successfully read out the Zigbee door sensor and increase the digital meter reading with every signal. What is missing now is feeding into InfluxDB and configuring Grafana to get nice evaluation graphs for desired time periods.

Also, in the future I want to work on even more exciting IOT stuff with cool use cases. I'm thinking about migrating some IOT processes from my Smart Life app to my NodeRED.

The Rasp4 itself offers even more possibilities than just being used as an IOT server. For example, I already use it as a DNS server with Pi-Hole. With this it blocks e.g. nervice DNS queries like telemetry data or advertisements. For sure I will find more cool use cases for my Rasp4.

Summary

IOT is an exciting topic. Ready-made solutions like Smart Life, Alexa and co are easy to install and promise early the desired success. However, such ready-made solutions are mostly very limited in functionality. So far, I have made good progress with Smart Life.

However, the task of reading the gas meter automatically has called for a more complex IOT solution with a Raspberry and NodeRED Flow. I described how this solution looks like here. I had an incredible amount of fun working on this IOT project and I am extremely grateful to my IOT buddy :D for spending the time to bring me closer to the IOT topic.

Did you find the article interesting or do you have any questions? Feel free to write me :) .

Become a Patreon!

ALPS Helper Library

Sat, 19 Dec 2020 00:00:00 GMT

Hi :).

The new ALPS Converter I developed makes it possible to convert an ALPS specification (spec short) to OpenApi, GraphQL or other APIs.

import { Alps, FormatType } from 'alps-unified-ts';

// loaded from a YAML file
Alps.unified(Alps.loadYaml('todo-alps.yaml'), { formatType: FormatType.OPENAPI })

Other great features of my ALPS unified library are:

automatic versioned releasing to NPM, PyPi, Maven and Nuget (for .NET).
Type support

The GitHub project were created using Projen. I will discuss what the ALPS Api and Projen is in more details in the next sections.

ALPS API

ALPS is a specification for describing the context of a service. ALPS can be used as specification input to generate low abstracted specifications like OpenApi / Swagger, WSDL, RAML, WADL.

The ALPS Converter converts an ALPS Api Spec into lower abstracted Api specifications. An example of such a conversion follows in the next section.

ALPS API Spec Example

The following example is a simple TODO ALPS API Spec.


alps:
  version: '1.0'
  doc:
    value: 'Simple Todo list example'

  ####################################
  # metadata
  ext:
    - type: metadata
      name: title
      value: simpleTodo
      tags: 'oas'
    - type: metadata
      name: id
      value: http://alps.io/profiles/mamund/simpleTodo
      tags: 'oas'
    - type: metadata
      name: root 
      value: http://api.example.org/todo
      tags: 'oas'
  
  descriptor:
    # properties
    # - these are the data elements
    - id: id
      type: semantic
      text: storage id of todo item
      
    - id: body
      type: semantic
      text: content of todo item

    # groupings
    # - these are the storage objects
    - id: todoItem
      type: group
      text: todo item
      descriptor:
      - href: '#id'
      - href: '#body'

    # actions
    # - these are the operations
    - id: todoList
      type: safe
      rt: todoItem
      text: return list of todo items
            
    - id: todoAdd
      type: unsafe
      rt: todoItem
      text: create a new todo item
      descriptor:
      - href: '#todoItem'
      
    - id: todoRemove
      type: idempotent
      tags: delete
      rt: todoItem
      text: remove a single todo item
      descriptor:
      - href: '#id'

The element todoItem consists of an id and a todo string body . Three actions are defined todoList to list the todo items, todoAdd to insert new todos and todoRemove to delete todo items.

From these, the Unified Convertor can generate this OpenApi specification:

openapi: 3.0.1

# *******************************************************************
# generated by "unified" from src/todo-alps.yaml
# date: Wed Nov 25 2020 18:47:05 GMT+0100 (Central European Standard Time)
# http://github.com/mamund/2020-11-unified
# *******************************************************************

info:
  title: simpleTodo
  description: Simple Todo list example
  version: 1.0.0

servers:
- url: 'http://api.example.org/todo'

paths:
  /todoList:
    get:
      summary: 'return list of todo items'
      operationId: todoList
      responses:
        200:
          description: todoList
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/todoItem'
  /todoAdd:
    post:
      summary: 'create a new todo item'
      operationId: todoAdd
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/todoItem'
      responses:
        200:
          description: add todoAdd
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/todoItem'
  /todoRemove/{id}:
    delete:
      summary: 'remove a single todo item'
      operationId: todoRemove
      parameters:
        - name: id
          in: path
          description: id of todoRemove
          required: true
          schema:
            type: string
      responses:
        204:
          description: delete todoRemove

components:
  schemas:
    todoItem:
      description: todo item
      type: object
      properties:
          id:
            type: string
            example: 4572
          body:
            type: string
            example: jn1ov7axj4kv560d0921xf

Or in GraphQL schema:

# *******************************************************************
# generated by "unified"
# date: Fri Dec 11 2020 16:51:00 GMT+0100 (Central European Standard Time)
# http://github.com/mamund/2020-11-unified
# *******************************************************************

type todoItem {
  id: String!
  body: String!
}

type Query {
  todoList: [todoItem]
}

type Mutation {
  todoAdd(todoItem: String!): todoItem
  todoRemove(id: String!): todoItem
}

schema {
  query: Query,
  mutation: Mutation
}

Projen

Projen allows sophisticated management of project configuration files through code. With just a few lines of TypeScript code, an entire repository can be configured. Here is an example:

const { JsiiProject } = require('projen');

const project = new JsiiProject({
  name: 'alps-unified-ts',
  authorAddress: 'damadden88@googlemail.com',
  authorName: 'Martin Mueller'
});

project.synth();

These few lines create all the GitHub project files your heart desires. Among them the package.json, .gitignore, tsconfig.json and many many more.

What's really cool is that Projen also comes with GitHub workflows that can publish new versions to registries like NPM or PYPI.

I fell in love with the Projen framework as it is a great abstraction and default for a setup of projects. I encourage you to give Projen a try and very soon you will experience the beauty of this framework yourself.

ALPS Library in Action

Here I would like to briefly introduce my new ALPS Unified Library. The TypeScript package can be used like this

import { Alps, FormatType } from 'alps-unified-ts';

// loaded from a YAML file
Alps.unified(Alps.loadYaml('./spec/todo-alps.yaml'), { formatType: FormatType.OPENAPI })

// or directly via TypeScript Object
Alps.unified(Alps.spec({
    alps: {
      version: '1.0',
      doc: {
        value: 'Simple Todo list example',
      },
      ...
    }
});

The Alps.unified(...) function always returns a string. Depending on the formType the returned string contains the desired format. In our example this is FormType.OPENAPI which is OpenApi in YAML. If you would now like to have OpenAPI returned in JSON, FormatType.OPENAPI_JSON would also work. For rendering the Graph QL schema you would have to use FormatType.DSL.

In Python, the library can be used like this:

import alps_unified_ts as alps

alps_def = alps.AlpsDef(
  version='1.0', 
  descriptor=[alps.DescriptorDef(id="id", type="semantic", text="sotrage id of todo item")], 
  doc=alps.DocDef(
    value="Simple Todo list example"), 
    ext=[
      alps.ExtDef(
        name="root", 
        tags="oas" 
        type="metadata", 
        value="http://api.example.org/todo"),
      alps.ExtDef(
        name="title", 
        tags="oas" 
        type="metadata", 
        value="simpleTodo")])

alps_converted = alps.Alps.unified(alps_document=alps.Alps.spec(alps=alps_def), format_type=alps.FormatType.OPENAPI)

Using it in Java and .NET should look similar.

In my previous blogpost ALPS combined with CDK, I explain how to optionally create an AWS Api Gateway or AWS Appsync from an ALPS Spec using the ALPS Todo example and the ALPS unified library.

Summary

Building a library with Projen was super fun! I used it to build a new, improved version of the alps unified library. It now comes with super cool new features that I introduced here.

By the way, one more info. If you are also interested in the ALPS topic, with the ALPS Community we regularly organize community meetings online. There you can meet exciting people from all over the world. Feel free to come by :) .

Become a Patreon!

ALPS API combined with AWS CDK

Mon, 14 Dec 2020 00:00:00 GMT

Hi :).

AWS offers many exciting Api technologies. Those include the AWS Api Gateway which is an implementation of a REST Api or AWS Appsync which implements a GRAPH QL Api.

Each of these AWS Api implementations has its advantages and disadvantages. For example, Appsync can be faster to build than the Api Gateway, but then again it gets more complicated when creating queries to the Graph QL. Admittedly, I don't know enough to weigh the pros and cons of the different Api implementations in AWS, but I don't have to anymore!

With my exciting work with ALPS in combination with AWS CDK, I can make the Api freely selectable or even interchangeable and all based on an ALPS specification (Spec for short). As AWS says itself. Choice Matter! :)

What exactly an ALPS Spec is and how Api implementations like Api Gateway or Appsync can be created from it, I will explain in the next sections.

ALPS Api

ALPS is a specification to describe the context of a service. ALPS can be used as specification input to generate low abstracted specifications like OpenApi / Swagger, WSDL, RAML, WADL.

When I saw the YouTube video with Mike Amundsen, I immediately thought the idea of ALPS was cool and exciting. Like any good developer I love abstractions and ALPS seems to be an extremely cool abstraction. I immediately had the idea to connect the ALPS Api with an AWS CDK construct. That's exactly what I did and I tell more in details about it in the next section.

AWS CDK ALPS Constructs

AWS CDK Constructs are, in short, source code for deploying resources or services in AWS. For this it uses abstract languages like TypeScript or Python and generates cloudformation templates from the code which are then also applied. If you are interested in the topic of AWS CDK, I have written a lot of posts about CDK here.

So my goal was to have targeted AWS Apis like Api Gateway and Appsync built using an ALPS specs. Below, I first show an ALPS Spec example. Then I describe the construct where I generate an AWS Api Gateway from an ALPS Spec. After that, an Appsync is built from the same ALPS Spec.

The two subsequent libraries are automatically released after NPM (for JavaScript and TypeScript) and PYPI (for Python). In the future I also plan to publish to a public Maven repository and .Net package repository.

ALPS Spec Example

The following example is a simple TODO list api.


alps:
  version: '1.0'
  doc:
    value: 'Simple Todo list example'

  ####################################
  # metadata
  ext:
    - type: metadata
      name: title
      value: simpleTodo
      tags: 'oas
    - type: metadata
      name: id
      value: http://alps.io/profiles/mamund/simpleTodo
      tags: 'oas'
    - type: metadata
      name: root 
      value: http://api.example.org/todo
      tags: 'oas'
  
  descriptor:
    # properties
    # - these are the data elements
    - id: id
      type: semantic
      text: storage id of todo item
      
    - id: body
      type: semantic
      text: content of todo item

    # groupings
    # - these are the storage objects
    - id: todoItem
      type: group
      text: todo item
      descriptor:
      - href: '#id'
      - href: '#body'

    # actions
    # - these are the operations
    - id: todoList
      type: safe
      rt: todoItem
      text: return list of todo items
            
    - id: todoAdd
      type: unsafe
      rt: todoItem
      text: create a new todo item
      descriptor:
      - href: '#todoItem'
      
    - id: todoRemove
      type: idempotent
      tags: delete
      rt: todoItem
      text: remove a single todo item
      descriptor:
      - href: '#id'

The todoItem element consists of an id and a todo string body . Three actions are defined todoList to list the todo items, todoAdd to insert new todos and todoRemove to delete todo items.

CDK ALPS REST Api Construct Library

The first ALPS CDK Construct I implemented is the CDK ALPS REST Api Construct in my github repo . It creates an AWS Api Gateway using the CDK SpecRestApi Construct. This then just needs to be passed an openApi Spec generated from the ALPS:

const api = new apigw.SpecRestApi(this, 'SpecRestApi', {
  apiDefinition: apigw.ApiDefinition.fromInline(oasSpecJSON),
});

CDK ALPS Graph QL Api Construct Library

The second ALPS CDK Construct was much more exciting for me, because I used AWS Appsync as Graph QL Api and it was completely new for me. But against all expectations it was very easy and I was able to finish the construct in a few hours here on github. The ALPS compiler translates the ALPS spec into the Graph QL schema and passes it to the CDK GraphqlApi Construct:

export class AlpsGraphQL extends appsync.GraphqlApi {

  constructor(scope: cdk.Construct, id: string, props: AlpsGraphQLProps) {
    // convert ALPS yaml to graph ql schema file in tmp/schema.graphql
    const sdl = Alps.unified(Alps.loadYaml(props.alpsSpecFile), { formatType: FormatType.SDL });
    fs.writeFileSync('tmp/schema.graphql', sdl);
    super(scope, id, {
      ...props,
      schema: appsync.Schema.fromAsset('tmp/schema.graphql'),
    });

    new cdk.CfnOutput(this, 'GraphQlUrl', { value: this.graphqlUrl });
  }
}

CDK Demo Deployment

Sweet! We now have these two constructs which can generate AWS Apis from an ALPS Spec. Now we just need to do that :). For this I created a new Demo Github Repo.

The demo deployment uses the same ALPS Spec TODO example I already presented. DynamoDB is used to store the TODO items:

const todoTable = new db.Table(stack, 'TodoTable', {
  removalPolicy: RemovalPolicy.DESTROY,
  partitionKey: {
    name: 'id',
    type: db.AttributeType.STRING,
  },
});

Then the Api Gateway and Appsync are created

new AlpsSpecRestApi(stack, 'AlpsSpecRestApi', { alpsSpecFile: 'src/todo-alps.yaml' });
...

const graphQlApi = new AlpsGraphQL(stack, 'AlpsGraphQL', { alpsSpecFile: 'src/todo-alps.yaml' });

...

That's it. Now just apply the CDK stack with:

yarn build
yarn deploy [--profile X]

Yeahh. So we have created an Api Gateway and an Appsync based on the ALPS Api.

Use Cases

Of course I thought about what use cases there could be for the ALPS spec abstraction. Some of them came to my mind while I was creating the CDK ALPS Constructs.

I see it as an easy and safe way to migrate from Api Gateway to Appsync. For example, the resolvers for the Graph QL could use the Lambda integrations from the Rest Api, or possibly even do away with them entirely and map directly to a DB for CRUD operations.

Since the ALPS spec is an abstraction of e.g. Rest Api, it should be possible for domain experts to write it themselves. It would be great if the domain specific context could be abstracted better and this could be possible with ALPS. So while the domain expert takes care of the domain specificity, the developer can work on the concrete api and its implementation. Thus, a better modular work is made possible.

Summary

ALPS Api is a fascinating idea about abstraction from other apis like REST, Graph QL etc. Personally, it helped me understand Graph QL better as I had a common denominator by going back through the ALPS spec. Also the auto-generation of Graph QL schema is super great to understand the api better as well.

I myself realized while creating the three repos that I still don't know much about the ALPS syntax and want to learn more about it. If you are also interested in the ALPS topic, please write me. With the ALPS Community we regularly organize community meetings online from all over the world. There you can meet exciting people and get involved if you want :).

I am already working on a refreshed library to convert the ALPS spec to the lower abstracted Apis here. This will make it even easier to use ALPS unified as a library in your code:

// loaded from a YAML file
Alps.unified(Alps.loadYaml(...), { formatType: FormatType.OPENApi })

// or directly via TypeScript object
Alps.unified(Alps.Spec({
    alps: {
      version: '1.0',
      doc: {
        value: 'Simple Todo list example',
      },
      ...
    }
});

In addition, the library should work in JavaScript, TypeScript, Python, Java and .NET and be available through public registries. More about this will come in a separate blog post. Until then stay tuned!

Become a Patreon!

Thankfulness

Tue, 08 Dec 2020 00:00:00 GMT

A warm welcome,

As we are now approaching the end of 2020, I would like to take this opportunity to show my gratitude. For this I would like to dedicate a whole blog post :). Even if it was different for many, for us 2020 was the best year as a family. Why this is so, you will find out when you read the whole post.

In the next sections I list my thanks chronologically by my life.

God

The first and greatest thanks goes to God. He is merciful and generous. He has blessed me with a healthy mind and body, which is by no means for granted. He has accompanied me on my path of life and saved me from greater harm.

Thank you God and I humbly ask you to continue to watch over me and especially over my family, friends and humanity. Just saying thanks is not enough for me! In 2021 I would like to become more involved in your church.

Even if it is not always easy, you give me the strength to do so. I want to thank you my God.

Parents

Many thanks also to my mom and dad. They are two incredibly great people. And above all they are kindhearted. They have given me great values. My childhood as an outsider was certainly not easy, but with the help and affection of my family it was no problem to get through this relatively difficult time of my life.

I would especially like to thank my mother, who was instrumental in my development. For her, education has always been a top priority for me and my brother. I am extremely grateful to her for this, because without this high value I might never have found my dream job as a software developer.

She is a kind and strong woman. I think this has influenced me in the choice of my wife who is also kind and strong. Thanks a lot mom and dad.

Woman

For the thanks to my wife. I don't know where to begin. I have so much to be grateful for when it comes to my wife. I got to know her in England. I was there for almost two years. After an internship at Alfresco, I got a permanent position there and decided to stay in England for a longer.

My wife was in England for almost a year. She is from Brazil and was eager to get to know England. When we met we immediately had a connection. Love at first sight!

Then everything went relatively quickly. We moved into an apartment together and got engaged only a few months after. Since then we have been together for 3 wonderful years. Our relationship is very harmonious and I am looking forward to the many years ahead of us.

Many thanks to my beautiful, smart and wonderful wife, for always being at my side and walking through the adventure of life with me. I don't want to be a day without you anymore. And also many thanks that you are such a great mother. Our daughter is so lucky that she has such a great mommy.

Child

A special thanks goes to my daughter. She is not yet able to understand this text. Let's see, maybe this blog still exists, if she can and she is perhaps happy about it. Anyway I am very proud and happy to be your father.

You give me your trust every day and I accept this precious gift every day and promise to always act for your best. Thank you so much for letting me be your father.

To all Others

Here I would like to thank everyone who deserves an own section but then the mail would be extremely long.

A special thanks goes to my brother. He played and plays an important role in my life. We grew up together and were best friends. We went through thick and thin together during kindergarten and sometimes even at school. Many thanks.

Many thanks also to my non core family like my aunts and uncles and my grandparents. We owe you so incredibly much. You have helped my wife and me a lot with the move to Germany. You have supported us so much in buying a house. Furthermore, the many small supports like clothes for Rebecca and the many small gifts were extremely helpful. Thanks a lot for your help.

I would also like to thank my family from Brazil. I had the luck to visit them in Brazil in 2018 and hope to do so again in 2021.

As well thank you to my former colleagues from Alfresco and Object. They have shaped me a lot and made me a better developer.

We have only recently started living in the beautiful city called Ludwigslust. Not far from us we have found a great young family with whom we like to spend time. They are kind people and loving parents. Thank you very much that you are here and that we can have you as friends.

At the very beginning I thanked God. I also want to thank God's church here in Ludwigslust. For me, the Landeskirchliche Gemeinschaft Ludwigslust (church community) is a valuable community with the goal of doing good and proclaiming God's word. Thank you very much for letting me be your member.

Summary

Gratitude is a great thing and makes life more beautiful. It creates a positive feeling for the person to whom the gratitude is directed and for the one who expresses it. I hope you liked this slightly different stem anyway. Write me what you are thankful for.

Become a Patreon!

Optimised Fasting

Sat, 28 Nov 2020 00:00:00 GMT

Hi Low Carber :)

In the last few months I have successfully lost a lot of weight with Optimised Fasting. Per week I have lost about 500 grams, that is half a kilo. I have already gained a lot of experience with fasting like water fasting. Optimised Fasting is a variation of water fasting. Since this diet works so successfully for me, I wanted to introduce it to you.

Disclaimer

You should only try this diet if you have some experience with low carb or even ketosis. Low carb means a massive reduction in the intake of any kind of sugar. The human being has essentially two metabolisms to produce energy. One is the sugar metabolism and the other is the fat metabolism. In sugar metabolism, the body primarily uses sugar to produce energy and in fat metabolism, the body primarily uses fat.

However, if the fat metabolism is hardly used, it atrophies or is never properly developed. With the Optimised Fasting we need a strongly trained fat metabolism, otherwise the two days of fasting can be physically and mentally difficult, because the body longs for sugar and has never learned to deal with its own body fat through fat metabolism. So it would be a bit like jumping into cold water. Side effects can then be headaches and extreme hunger.

So if you think you don't have a well developed fat metabolism, I recommend to try a low carb diet and to experience a day of water fasting at the weekend. Here is still an older post of me over the water chamfered which I can recommend you warmly.

In the next sections I would like to tell you more about my new diet form of the Optimised Fasting. With this it concerns primarily to decrease over many weeks continuously with minimum expenditure.

Optimised Fasting instructions

Optimised Fasting consists of approx. 2 days long water fasting. Around the training it is allowed to add vitamins, salts and minimal proteins. By the supply of protein before and after the training, an anabolic signal for the body is virtually released, which helps to carry out the training better and above all more intensive.

The amount of protein must not be too high, otherwise the digestive system is stressed too much and this becomes a hindrance to training. About 25g before and after training should be enough.

The vitamins are generally important for humans to maintain the body functions. The body can store little or no salt in the body and therefore it is necessary to replenish salt during fasting. The easiest way to do this is to consume a food with a high salt content.

An exact example of the foods I eat before and after training is listed in the next chapter.

The goal is to continue fasting until you have reached 500 grams less than last week. For me this was always the third day, the day after the two days of Optimised Fasting. When I have reached 500 grams, I always make a picture of the dare with my smartphone. So I know then how many KG I want to reach next week.

Optimised Fasting Example

I work as a software developer for 4 days a week. So from Monday to Thursday. Therefore I start the Optimised Fasting already on Thursday. Wednesday evening is the last time I eat something.

Already on Wednesday I reduce food with high sugar content. I then mainly eat only protein and fat. This helps me to prepare my fat metabolism for the next 2 days of fasting and makes it easier to start.

Thursday and Friday we fast and only eat a little something for my morning training.

Before the Training

I make myself a green tea with a hot chilli sauce and pressed lemon juice. I hope that this will have a stimulating effect and burn more calories due to the increased caffeine content and spicy substances in the chili. Afterwards it goes to the training.

My training in the morning is broad and very varied. Depending on my body condition I do some strength training, endurance training or a short high intensity training. Warming up and subsequent stretching is also part of my training program. If you want to know more about my training, let me know :).

After the Training

I eat a small bowl full of nuts (walnut, pumpkin seeds, Brazil nut), Omega 3 and vitamin D capsules, a piece of black chocolate, gummi bears (about 8) and two eggs. In the cover picture you can also see this meal. If you are wondering why the gummi bears. They contain a lot of gelatine which is a healthy and important protein for the body. They are also delicious and the sugar in them is too little to get me out of ketosis.

What I find so great about Optimised Fasting

Any diet is only successful if it is easy to implement and Optimised Fasting is very easy to implement. It must be fasted only two days and on the remaining days one can eat as one would like. There is no need for time-consuming preparation of meals, because you are fasting.

When water fasting, I have always been annoyed by drinking a lot of salt water. By Optimised Fasting enough salt can be supplied to the body by the snack after training. It is then no longer necessary to drink extra salt in the form of salt water.

I also find it cool that the 500 grams I reach every weekend is a very reliable value, since my bowels are mostly empty after two days of fasting and the water, which was caused by sugar deposits in the muscles, is out. Thus I am subject to fewer water fluctuations during the weight admission after the second chamfering day.

Simplified Entrance

The implementation of my previous Optimised Fasting was very easy for me because I am used to fasting and I generally eat low carb. I am also very disciplined when it comes to sports and I usually do short and crisp workouts in the morning and evening. But I can imagine that you might find it a bit harder to get started.

For this case I have thought of a simplified variant. Start the first two weeks with an Optimised Fasting Day. This gives your body the opportunity to get used to the fasting slowly. Maybe one fasting day is enough for you and the next day you have already lost 500 grams of weight compared to last week. That would be great :).

Summary

Optimised Fasting is an easy way to lose weight continuously. It is extremely easy to implement since only about two days must be chamfered and in the remaining week you can eat what you want. No annoying preparation of the meals. No counting the calories.

If fasting or low carb in general is something for you, I can only recommend you to try this kind of weight reduction. Write me your experiences. I am very curious :) !

Become a Patreon!

Alfresco 6.2 with OCR

Tue, 24 Nov 2020 00:00:00 GMT

Hi Alfrescans.

Object Character Recognition (short OCR) is a cool tool as it allows you to make your scanned documents incredible useful as it recognizes characters and words and makes those accessible in your document like through a PDF sandwich. A PDF sandwich contains text which you can select, copy or search through it. That's the case like when you export a word document into PDF.

But when you just scan a text and save it as PDF, the text is not selectable anymore and you only have some less useful pictures in your PDF document. And here the power of OCR comes in. With OCR you can transform those dumb PDF document to PDF sandwich. Than you can again select and copy text or search for it in the document.

Even better as Alfresco uses Solr to make a full text search, you can search for documents containing certain words across you documents. Thats extremely powerful. But yeah Alfresco doesn't come ootb with OCR so how you can enhance Alfresco with such a cool feature? In the next sections I will explain how!

OCR as a Service

As a quick way of developing, I implemented my solution as a Docker Compose deployment. The OCR service itself is implemented as a Docker container which shares the ocr input and output volumes with the Alfresco container. The underlying OCR engine is OCRmyPDF.

ocrmypdf:
    build: ocrmypdf
    hostname: ocrmypdf
    mem_limit: 512m
    volumes:
        - ocr-input:/usr/local/tomcat/ocr_input
        - ocr-output:/usr/local/tomcat/ocr_output

The full Docker Compose git code can be found in my GitHub repo alfresco-ocr. I describe in the next section how I did the setup with Alfresco.

Alfresco Setup

The Alfresco Docker Compose deploy is created using the Alfresco Docker Installer with the ACS Community 6.2 option. All other options are the defaults.

Additionally the repo and share OCR amps alfresco-simple-ocr are installed by simply dropping the addon jar files into alfresco-ocr/alfresco/modules/jars and alfresco-ocr/share/modules/jars . Easy peasy.

Run it with AWS CDK

I enhanced the Docker Compose deployment to make it easy runnable in AWS. For that I use AWS CDK. Is use a little library which hides a lot complexity using the AWS CDK stuff. The CDK Pipeline App in ./cdk/app.ts will create a sophisticated pipeline to deploy the instance in one or more accounts. For more information please read the Readme.

Your Alfresco will become available in AWS und links like

http://ec2-3-92-227-231.compute-1.amazonaws.com:8080/ for Alfresco Content App or
http://ec2-3-92-227-231.compute-1.amazonaws.com:8080/share for Share of course

I personally like that way of deploying Alfresco into the Cloud as AWS CDK is a lovely abstraction on top of CloudFormation which makes creating infrastructure such as for Alfresco (Ec2, Loadbalancer, Kubernetes, Docker) a piece of cake. If you want to know more about deploying Alfresco into AWS have a look through my posts where you find many more topics around it.

Summary

OCR in Alfresco is a cool and useful feature. It enhances an Alfresco repository to make better use of documents which lost text information like scanned ones. It's incredibly useful it is to encapsulate the OCR engine as its own service like I did here as Docker Container.

Potentially you could enhance that setup to offer a REST API to the OCR service and for example choose between difference OCR engines and pick the best result yourself. All that is very exciting. Tell me what you think about OCR together with Alfresco.

Become a Patreon!

AWS Cognito Auth Mock

Mon, 16 Nov 2020 00:00:00 GMT

Ahoi AWS'ler

In my AWS project, where I can start, stop and schedule instances of Alfresco via REST API, I use AWS Cognito as identity provider for user management. Cognito makes it easy for me to create new users and give them the rights to access the REST API.

The REST API is implemented using AWS API Gateway and a Cognito Authorizer allows users from the Cognito Identity Pool to access the endpoints.

The Cognito Authorizer is especially important because I have implemented a permission layer that prevents users from manipulating Alfresco instances of other users. Basically the Permission Layer works with the User ID returned by the Cognito Authorization process if the authorization was successful.

This works great, but I would like to have automated tests to test the functionality of the permission layer. It would be great if I could use Postman for this. Unfortunately this is not possible because the authentication process with Cognito requires user interaction to retrieve the credentials.

The solution for me was to write a Mock Authentication Layer which simulates the exact behavior of the Cognito Authorizer. For this I used Middy. What Middy is and how exactly my solution looks like I will describe in the next sections.

Middy as Mock Auth Layer

Middy is a simple framework for building layers in Lambda functions. With this framework it is possible to encapsulate all features that are not directly related to business logic in layers. In my case I want to implement a Mock Auth Layer that simulates the Cognito authorization. This layer should be optionally switchable by environmental variables.

The complete code for the Auth Layer can be found on GitHub.

const mockAuthLayer = (config?: MockAuthLayerConfig) => {
  return {
    before: (handler: any, next: () => void) => {
      handler.event = handler.event ?? {};
      handler.event.headers = handler.event.headers ?? {};
      const mockHeaderPrefix = config?.mockHeaderPrefix || 'MOCK_AUTH_';
      Object.keys(handler.event.headers)
        .filter((headerKey) => {
          return headerKey.startsWith(mockHeaderPrefix);
        })
        .forEach((headerKey) => {
          const headerValue = handler.event.headers[headerKey] || 'martin';
          handler.event.requestContext = handler.event.requestContext ?? {};
          handler.event.requestContext.authorizer = handler.event.requestContext.authorizer ?? {};
          handler.event.requestContext.authorizer.claims = handler.event.requestContext.authorizer.claims ?? {};
          handler.event.requestContext.requestContext.authorizer.claims[
            headerKey.substring(mockHeaderPrefix.length).replace(/&/g, ':')
          ] = headerValue;
          console.log(`shifted header ${JSON.stringify(handler)}`);
        });
      next();
    },
  };
};

Ok let's have a look at the code.

const mockHeaderPrefix = config?.mockHeaderPrefix || 'MOCK_AUTH_';
  Object.keys(handler.event.headers)
    .filter((headerKey) => {
      return headerKey.startsWith(mockHeaderPrefix);
      ...

So I filter all headers that start with a certain prefix. If none is set I use MOCK_AUTH.

const headerValue = handler.event.headers[headerKey] || 'martin';
  handler.event.requestContext = handler.event.requestContext ?? {};
  handler.event.requestContext.authorizer = handler.event.requestContext.authorizer ?? {};
  handler.event.requestContext.authorizer.claims = handler.event.requestContext.authorizer.claims ?? {};
  handler.event.requestContext.requestContext.authorizer.claims[
    headerKey.substring(mockHeaderPrefix.length).replace(/&/g, ':')
  ] = headerValue;

Then the headers are copied into the Authorizer Claims area which would also contain the Cognito Values. This way my Lambda has access to the authenticated user and role. How to set the MOCK_AUTH_ headers is shown in the next section.

At the end with .replace(/&/g, ':') there is a small symbol replacement because Postman does not allow to use double colons : in the header key.

Lambda Unit Tests

For my Lambdas I have written unit tests which should also take the authorization into account. From my code I pick a unit test file on GitHub and explain it in more detail.

...
it('from himself will success', async (done) => {
  awsSdkPromiseResponse.mockReturnValueOnce({ Items: [{ instanceId: 'i123', userId: 'martin' }] });
  await handler(
    {
      headers: {
        'MOCK_AUTH_cognito:username': 'martin',
        'MOCK_AUTH_cognito:groups': 'Admin',
      },
      queryStringParameters: { userId: 'martin' },
    },
    {} as Context,
    (_, result) => {
      expect(result?.statusCode).toBe(200);
      ...

So as we can see here I use the mock header MOCK_AUTH_cognito:username': 'martin' and 'MOCK_AUTH_cognito:groups': 'Admin' . With this I simply simulate the authorization behavior of Cognito and make the username and user groups known to the Lambda at runtime. Fair enough.

Postman Tests

Of course my Postman tests should also be able to use the new Auth Mock Layer. On GitHub I have a Postman Collection with over 40 requests. The whole collection is executed by Newman after building the DEV environment as an integration test:

  ...
  testCommands: (stageAccount) => [
    ...(stageAccount.stage === "dev"
      ? [
          `npx newman run test/alf-cdk.postman_collection.json --env-var baseUrl=$RestApiEndPoint -r cli,json --reporter-json-export tmp/newman/report.json --export-environment tmp/newman/env-vars.json --export-globals tmp/newman/global-vars.json; RESULT=$? || \,
          ./scripts/cleanup.sh
          exit $RESULT`,
        ]
      : []),
  ],
  ...

To use the Mock Auth Layer you just have to set the headers, like you did with the unit tests in the previous section:

MOCK_AUTH_cognito&username = martin
MOCK_AUTH_cognito&groups = Admin

But I had to make one change. Postman does not allow to have a double colon in the header key. So I had to add a symbol replacement. The AND character & is translated to the double colon : in the code.

If you are interested in testing with CDK, I can recommend my blog post from last week.

Summary

AWS Cognito is mega cool and fast to set up. It offers a range of great user management functions. Unfortunately, it is not possible to run the Cognito authorization process completely automated, e.g. for automated tests. But that doesn't matter because I simply simulate the Cognito authorization with my Auth Mock Layer. Now to you. Did you like the article? Do you want to know more? Let me know.

Become a Patreon!

AWS CDK - PipelineApp Library - Automated Testing

Mon, 09 Nov 2020 00:00:00 GMT

Hi CDK Fans,

Recently I published a blog post about the AWS CDK PipelineApp Library I'm working on. If you are interested in the AWS CDK Pipeline you should read that one! Meanwhile I have used the library dozens of times in my projects and have developed it further.

In this post I would like to explain how to use my library to run automated tests with the pipeline.

By the way, my CDK Library can be used directly via npm dependency and does not require an npm repository. Just specify the dependency as follows:

 "dependencies": {
    "alf-cdk-app-pipeline": "github:mmuller88/alf-cdk-app-pipeline#v0.0.8
    ...
 }

What are we testing for?

In the age of fast releases to production, it is essential to perform automated tests during the release process in the pipeline. This ensures that old and new features still work.

Cloudformation stack testing

In the last blogpost I already introduced my Ec2 stack. Now I would like to go into more detail about the test part in the testCommands property.

const pipelineAppProps: PipelineAppProps = {
  branch: 'master',
  ...
  testCommands: (stageAccount) => [
    `curl -Ssf $InstancePublicDnsName
    aws cloudformation delete-stack --stack-name itest123 --region ${stageAccount.account.region}`,
  ],
};

With the command curl -Ssf $InstancePublicDnsName I test if the Ec2 instance is started successfully. Afterwards a cleanup is performed using the AWS CLI. Because the commands are passed as a list of strings string[] and are processed sequentially, the cleanup can take place after the curl command before the test days process the next command or return a fail.

The variable $InstancePublicDnsName was defined in the Ec2 stack and explicitly rendered into the test command. How it was defined you can see here:

const instancePublicDnsName = new CfnOutput(this, 'InstancePublicDnsName', {
  value: instance.instancePublicDnsName
});
this.cfnOutputs['InstancePublicDnsName'] = instancePublicDnsName;

API GW Tests

In my AWS API GW stack I run Postman tests. There are now more than 40 requests running against the API GW and testing the backend extensively. If you are interested in Postman Tests you can find a lot of articles about the topic on my blog page on https://martinmueller.dev/tags/postman .

const pipelineAppProps: PipelineAppProps = {
  branch: 'master',
  ...
  testCommands: (stageAccount) => [
    ...(stageAccount.stage === 'dev'? [
      `npx newman run test/alf-cdk.postman_collection.json --env-var baseUrl=$RestApiEndPoint -r cli,json --reporter-json-export tmp/newman/report.json --export-environment tmp/newman/env-vars.json --export-globals tmp/newman/global-vars.json
      aws cloudformation describe-stacks --query "Stacks[?Tags[?Key == 'alfInstanceId'][]].StackName" --region ${stageAccount.account.region} --output text |
      awk '{print $1}' |
      while read line;
      do aws cloudformation delete-stack --stack-name $line --region ${stageAccount.account.region};
      done`,
    ] : []),
  ],
};

With the CLI tool newman the Postman Collection, which contains more than 40 requests tests, can be executed. From aws cloudformation ... on there is also a cleanup of all remaining cloudformation stacks.

Summary

It feels like my AWS CDK pipeline library is becoming more useful and robust. With every new project of mine I add something to it and so it becomes a bit more sophisticated. I am very excited to see where the journey with my library will lead.

Become a Patreon!

AWS CDK - PipelineApp - A Library for Staging Pipelines

Sat, 24 Oct 2020 00:00:00 GMT

Hi CDK Fans,

Building AWS CDK pipelines is fun. Since the middle of this year (2020) there is even a high level CDK pipeline which brings many advantages like self-mutate, a simplified cross-account deployment and a better abstraction from the required code build projects.

I have developed a CDK Library which solves some annoying problems. If you encounter only one of the following, you should read the whole post :) :

Maintaining CDK dependencies between multiple repositories
Repetition of code for building CDK apps only to deploy e.g. an arbitrary stack
Repetition of CDK pipeline code
Missing a uniform CDK pipeline standard over e.g. an interface

In the next sections I will talk about this general library and how I solve those problems. First of all I will describe why you need a pipeline at all. Then I explain the idea behind the library and finally I show some examples of my projects which already use the library.

By the way my CDK Library can be used directly via npm depedency and does not require an npm repository. Just specify the dependency as follow:

 "dependencies": {
    "alf-cdk-app-pipeline": "github:mmuller88/alf-cdk-app-pipeline#v0.0.7
    ...
 }

Why a Pipeline at all?

In times of modern DevOps practices it is important to make changes in production as fast and easy as possible. Ideally this is done based on git commits and a staging pipeline. A Staging Pipeline executes changes on Stages.

In my company we have four stages DEV, QA, PROD. The DEV Stage is intended as a development environment for the developers. The QA Stage is a test environment that is as similar as possible to the PROD environment. And PROD itself is of course the production environment which is actively used by the company and our customers.

For a better understanding I will use only one DEV, PROD Stage in the next sections. The PipelineApp library can be extended by any number of stages.

Requirements

My CDK Pipeline Library should fulfill some requirements which are implemented externally via an interface. Here the interface is implemented with the PipelineAppProps for the PipelineApp class in the library:

const pipelineAppProps: PipelineAppProps = {
  branch: 'master',
  repositoryName: 'alf-cdk-ui',
  stageAccounts: [
    {
      account: {
        id: '123...',
        region: 'eu-central-1',
      },
      stage: 'dev',
    },
    {
      account: {
        id: '987...',
        region: 'eu-central-1',
      },
      stage: 'prod',
    },
  ],
  buildAccount: {
    id: '555...',
    region: 'eu-central-1',
  },
  customStack: (scope, account) => {

    const alfCdkSpecifics = {
      ...(account.stage === 'dev' ? {
        hostedZoneId: 'Z08...',
        domainName: 'i.dev.alfpro.net',
        certArn: 'arn:aws:acm:eu-central-1:123...:certificate/d40cd852-5bbf-4c1d-9a18-2d96e5307b4c',
      }
       : // prod
      {
        hostedZoneId: 'Z003...',
        domainName: 'i.alfpro.net',
        certArn: 'arn:aws:acm:eu-central-1:987...:certificate/4fe684df-36da-4516-bd01-7fcc22337dff',
      })
    }

    return new AlfCdkEc2Stack(scope, `${name}-${account.stage}`, {
      env: {
        account: account.id,
        region: account.region,
      },
      gitRepo: 'alf-ec2-1',
      tags,
      customDomain: {
        hostedZoneId: alfCdkSpecifics.hostedZoneId,
        domainName: alfCdkSpecifics.domainName,
        certArn: alfCdkSpecifics.certArn,
      },
      stackName: 'itest12',
      stage: account.stage,
    })
  },
  testCommands: (_) => [
    'sleep 240',
    `curl -Ssf $InstancePublicDnsName && aws cloudformation delete-stack --stack-name itest123 --region ${account.region}`,
  ],
};

// tslint:disable-next-line: no-unused-expression
new PipelineApp(pipelineAppProps);

In the next subsections I will explain the individual properties in more detail.

Git Repository

The PipelineApp will deploy its stages based on Git repositories. Therefore it is necessary to define the repository and the branch:

const pipelineAppProps: PipelineAppProps = {
  branch: 'master',
  repositoryName: 'alf-cdk-ui',
  ...

The repositoryName refers to the name in my repository on Github e.g. https://github.com/mmuller88/alf-cdk-ui . Via the pipeline this repository will be pulled by a token and defined as source.

Manage Stages

It should be possible to connect to specific AWS accounts on the respective stage. The specification about the account and the stage should be given as parameter. I developed the following interface:

stageAccounts: [
  {
    account: {
      id: '123..',
      region: 'eu-central-1',
    },
    stage: 'dev',
  },
  {
    account: {
      id: '987...',
      region: 'eu-central-1',
    },
    stage: 'prod',
  },
],

A list of stage accounts is passed to which the pipeline will deploy the stacks. Additionally the stage property defines the name of the stage. The order of the accounts also determines the order in which the staging is performed. In this example the DEV Stage is run through first and then the PROD Stage.

The build account defines the account where the CDK pipeline will be deployed.

buildAccount: {
  id: '555...',
  region: 'eu-central-1',
},

In the build account you can also define important secrets with the Secret Manager or the parameter store like the GitHub token. In the pipeline stack these secrets can be accessed there.

One or more Stacks

The pipeline should be able to deploy and integrate one or more stacks. The stack definition should be located in the respective repository and can then simply be passed to the Library as a Higher Order Function:

customStack: (scope, account) => {

    const alfCdkSpecifics = {
      ...(account.stage === 'dev' ? {
        hostedZoneId: 'Z08...',
        domainName: 'i.dev.alfpro.net',
        certArn: 'arn:aws:acm:eu-central-1:123...:certificate/d40cd852-5bbf-4c1d-9a18-2d96e5307b4c',
      }
       : // prod
      {
        hostedZoneId: 'Z00371764UBVAUANTU0U',
        domainName: 'i.alfpro.net',
        certArn: 'arn:aws:acm:eu-central-1:987...:certificate/4fe684df-36da-4516-bd01-7fcc22337dff',
      })
    }

    return new AlfCdkEc2Stack(scope, `${name}-${account.stage}`, {
      env: {
        account: account.id,
        region: account.region,
      },
      gitRepo: 'alf-cdk-ec2',
      tags,
      customDomain: {
        hostedZoneId: alfCdkSpecifics.hostedZoneId,
        domainName: alfCdkSpecifics.domainName,
        certArn: alfCdkSpecifics.certArn,
      },
      stackName: 'itest12',
      stage: account.stage,
    })
  },

The stack shown here is located in the alf-cdk-ec2 repo with alf-cdk-ec2-stack.ts as file. The stack is called in app.ts along with the PipelineApp Library.

Thanks to the High Order Function, account information can easily be integrated as a parameter into the respective Stage Stack using account.

Library Examples

In this section, I will show some of the consumers of the PipelineApp library.

Alf CDK UI React Frontend

The first example is about a frontend stack which builds a static website. The technologies I use here are React in TypeScript Flavor. The static site is hosted in a S3 bucket with a cloud front distribution.

All code is on GitHub alf-cdk-ui. The root directory contains the React App. The cdk folder contains all relevant CDK parts. Among them the UIStack in ui-stack.ts Construct and the PipelineApp in app.ts. The PipelineApp is called as follows:

import { UIStack } from './ui-stack';
import { name } from './package.json';
import { PipelineApp } from 'alf-cdk-app-pipeline/pipeline-app';
import { sharedDevAccountProps, sharedProdAccountProps } from 'alf-cdk-app-pipeline/accountConfig';


// tslint:disable-next-line: no-unused-expression
new PipelineApp({
  branch: 'master',
  repositoryName: name,
  stageAccounts: [
    {
      account: {
        id: '123...',
        region: 'eu-central-1',
      },
      stage: 'dev',
    },
    {
      account: {
        id: '123...',
        region: 'us-east-1',
      },
      stage: 'prod',
    },
  ],
  buildAccount: {
    id: '123...',
    region: 'eu-central-1',
  },
  customStack: (scope, account) => {
    const stageProps = {
      ...(account.stage === 'dev' ? {
        domainName: sharedDevAccountProps.domainName,
        acmCertRef: sharedDevAccountProps.acmCertRef,
        subDomain: sharedDevAccountProps.subDomain,
        hostedZoneId: sharedDevAccountProps.hostedZoneId,
        zoneName: sharedDevAccountProps.zoneName,
      } : { // prod stage
        domainName: sharedProdAccountProps.domainName,
        acmCertRef: sharedProdAccountProps.acmCertRef,
        subDomain: sharedProdAccountProps.subDomain,
        hostedZoneId: sharedProdAccountProps.hostedZoneId,
        zoneName: sharedProdAccountProps.zoneName,
      })
    };
    // console.log('echo = ' + JSON.stringify(account));
    return new UIStack(scope, `${name}-${account.stage}`, {
      stackName: `${name}-${account.stage}`,
      stage: account.stage,
      domainName: stageProps.domainName,
      acmCertRef: stageProps.acmCertRef,
      subDomain: stageProps.subDomain,
      hostedZoneId: stageProps.hostedZoneId,
      zoneName: stageProps.zoneName,
    })
  },
  buildCommand: 'make distcdk',
  manualApprovals: (account) => {
    return account.stage === 'dev' ? false : true;
  },
  testCommands: (_) => [
    // Use 'curl' to GET the given URL and fail if it returns an error
    'curl -Ssf $domainName',
    'echo done!!!',
  ],
});

As you can see here I use the same account for the two stages dev and prod. It would also be possible to use different accounts. You only have to make sure that the CDK bootstrap command trusts the respective account! Interesting is that I overwrite the default buildCommand here. The motivation for this is to fine-tune the React App during build to get it faster.

The most interesting part is the customStack property and the High Order Function. Here you can define the stack which should be managed by the PipelineApp. That's pretty cool because it allows me to keep the stack definition in the UI repo but I only have to import minimal code for the pipeline app library.

This is an important point for stacks like the UIStack. These are only allowed the aws-cdk dependencies like

import { StackProps, Construct, CfnOutput, RemovalPolicy } from '@aws-cdk/core';

only indirectly from the PipelineApp Library. This is more of an advantage than a disadvantage because only the aws-cdk dependencies have to be managed in the pipeline app library itself. And with a reasonable tagging strategy you can avoid aws-cdk update problems!

The build command can optionally be overwritten. Here I do this for example with make distcdk . In the projects I use Makefiles because they are excellent for encapsulating commands. With Makefiles I also have a nice overview of the possible and desired commands.

At the end I do a reachability test with the testCommands property. The parameter $domainName was defined in the stack like this:

this.domainName = new CfnOutput(this, 'domainName', {
  value: route.domainName,
});
this.cfnOutputs['domainName'] = this.domainName;

This notation must be followed so that the PipelineApp Library can retrieve the stack output variables after stack deployment.

Alf CDK Ec2 Backend

The next example is in Github on alf-cdk-ec2. Here I create an Alfresco deployment based on Docker Compose, Ec2 and CDK as infrastructure orchestrator. You can read more in the blog post ACS infrastructure creation made easy with AWS CDK.

const pipelineAppProps: PipelineAppProps = {
  branch: 'master',
  repositoryName: name,
  stageAccounts: [
    {
      account: {
        id: '123...',
        region: 'eu-central-1',
      },
      stage: 'dev',
    },
  ],
  buildAccount: {
    id: '123...',
    region: 'eu-central-1'
  },
  customStack: (scope, account) => {
    // console.log('echo = ' + JSON.stringify(account));
    const tags = JSON.parse(process.env.tags || '{}');

    const alfCdkSpecifics = {
      ...(account.stage === 'dev' ? {
        hostedZoneId: process.env.hostedZoneId || 'Z08...',
        domainName: process.env.domainName || 'i.dev.alfpro.net',
        certArn: process.env.certArn || 'arn:aws:acm:eu-central-1:123...:certificate/d40cd852-5bbf-4c1d-9a18-2d96e5307b4c',
      }
       : // prod
      {
        hostedZoneId: process.env.hostedZoneId || 'Z003...',
        domainName: process.env.domainName || 'i.alfpro.net',
        certArn: process.env.certArn || 'arn:aws:acm:us-east-1:123...:certificate/09d5c91e-6579-4189-882b-798301fb8fba',
      })
    };

    return new AlfCdkEc2Stack(scope, `${name}-${account.stage}`, {
      env: {
        account: account.id,
        region: account.region,
      },
      gitRepo: process.env.gitRepo || 'alf-cdk-ec2',
      tags,
      customDomain: {
        hostedZoneId: alfCdkSpecifics.hostedZoneId,
        domainName: alfCdkSpecifics.domainName,
        certArn: alfCdkSpecifics.certArn,
      },
      stackName: process.env.stackName || `itest123`,
      stage: account.stage,
    })
  },
  testCommands: (account) => [
    `aws ec2 get-console-output --instance-id $InstanceId --region ${account.region} --output text`,
    'sleep 180',
    `curl -Ssf $InstancePublicDnsName && aws cloudformation delete-stack --stack-name itest123 --region ${account.region}`,
  ],
};

// tslint:disable-next-line: no-unused-expression
new PipelineApp(pipelineAppProps);

In this example only one staging account is defined

Alf CDK Backend

The next example is also in Github alf-cdk. It is a very large stack consisting of Cognito, API GW, Lambdas, StepFunction, DynamoDB and more. This stack also creates other CDK stacks using a Lambda function.

import { name } from './package.json';
import { PipelineApp, PipelineAppProps } from 'alf-cdk-app-pipeline/pipeline-app';
import { AlfInstancesStack, AlfInstancesStackProps, alfTypes } from './lib/alf-instances-stack'
import { sharedDevAccountProps, sharedProdAccountProps } from 'alf-cdk-app-pipeline/accountConfig';

const pipelineAppProps: PipelineAppProps = {
  branch: 'master',
  repositoryName: name,
  stageAccounts: [
    {
      account: {
        id: '123...',
        region: 'eu-central-1',
      },
      stage: 'dev',
    },
    {
      account: {
        id: '987...',
        region: 'us-east-1',
      },
      stage: 'prod',
    },
  ],
  buildAccount: {
    id: '123...',
    region: 'eu-central-1'
  },
  customStack: (scope, account) => {
    // values that are differs from the stages
    const alfCdkSpecifics = {
      ...(account.stage === 'dev' ? {
        domain: {
          domainName: `api.${sharedDevAccountProps.zoneName.slice(0,-1)}`,
          zoneName: sharedDevAccountProps.zoneName,
          hostedZoneId: sharedDevAccountProps.hostedZoneId,
          certificateArn: `arn:aws:acm:us-east-1:${account.id}:certificate/f605dd8c-4ae3-4c1b-9471-4b152e0f8846`
        },
        createInstances: {
          enabled: true,
          imageId: 'ami-0ea3405d2d2522162',
          minutes: 5,
          maxPerUser: 2,
          maxInstances: 3,
          domain: {
            domainName: `i.${sharedDevAccountProps.zoneName.slice(0,-1)}`,
            hostedZoneId: 'Z0847928PFMOCU700U4U',
            certArn: `arn:aws:acm:eu-central-1:${account.id}:certificate/d40cd852-5bbf-4c1d-9a18-2d96e5307b4c`,
          }
        },
        swagger: {
          domain: {
            domainName: sharedDevAccountProps.domainName,
            certificateArn: sharedDevAccountProps.acmCertRef,
          }
        },
        auth: undefined,
      } : { // prod stage
        domain: {
          domainName: `api.${sharedProdAccountProps.zoneName.slice(0,-1)}`, // 'api.alfpro.net',
          zoneName: sharedProdAccountProps.zoneName,
          hostedZoneId: sharedProdAccountProps.hostedZoneId,
          certificateArn: `arn:aws:acm:us-east-1:${account.id}:certificate/62010fca-125e-4780-8d71-7d745ff91789`
        },
        createInstances: {
          enabled: false,
          imageId: 'ami-01a6e31ac994bbc09',
          minutes: 45,
          maxPerUser: 2,
          maxInstances: 50,
          domain: {
            domainName: `i.${sharedProdAccountProps.zoneName.slice(0,-1)}`,
            hostedZoneId: 'Z00371764UBVAUANTU0U',
            certArn: `arn:aws:acm:eu-central-1:${account.id}:certificate/4fe684df-36da-4516-bd01-7fcc22337dff`,
          }
        },
        swagger: {
          domain: {
            domainName: sharedProdAccountProps.domainName,
            certificateArn: sharedProdAccountProps.acmCertRef,
          }
        },
        auth: {
          cognito: {
            userPoolArn: `arn:aws:cognito-idp:us-east-1:${account.id}:userpool/us-east-1_8c1pujn9g`,
            scope: 'aws.cognito.signin.user.admin'
          }
        },
      }),
    }
    // console.log('echo = ' + JSON.stringify(account));
    const alfInstancesStackProps: AlfInstancesStackProps = {
      environment: account.stage,
      env: {
        region: account.region,
        account: account.id
      },
      stage: account.stage,
      stackName: `${name}-${account.stage}`,
      domain: alfCdkSpecifics.domain,
      createInstances: {
        enabled: alfCdkSpecifics.createInstances.enabled,
        imageId: alfCdkSpecifics.createInstances.imageId,
        alfTypes,
        automatedStopping: {
          minutes: alfCdkSpecifics.createInstances.minutes
        },
        allowedConstraints: {
          maxPerUser: alfCdkSpecifics.createInstances.maxPerUser,
          maxInstances: alfCdkSpecifics.createInstances.maxInstances,
        },
        domain: alfCdkSpecifics.createInstances.domain,
      },
      executer: {
        rate: 'rate(1 minute)'
      },
      swagger: {
        file: 'templates/swagger_validations.yaml',
        domain: {
          domainName: alfCdkSpecifics.swagger.domain.domainName,
          subdomain: 'openapi',
          certificateArn: alfCdkSpecifics.swagger.domain.certificateArn,
        }
      },
      auth: alfCdkSpecifics.auth,
    };

    return new AlfInstancesStack(scope, `${name}-${account.stage}`, alfInstancesStackProps);
  },
  manualApprovals: (account) => {
    return account.stage === 'dev' ? false : true;
  },
  testCommands: (account) => [
    ...(account.stage==='dev'? [
      `npx newman run test/alf-cdk.postman_collection.json --env-var baseUrl=$RestApiEndPoint -r cli,json --reporter-json-export tmp/newman/report.json --export-environment tmp/newman/env-vars.json --export-globals tmp/newman/global-vars.json`,
      'echo done! Delete all remaining Stacks!',
      `aws cloudformation describe-stacks --query "Stacks[?Tags[?Key == 'alfInstanceId'][]].StackName" --region ${account.region} --output text |
      awk '{print $1}' |
      while read line;
      do aws cloudformation delete-stack --stack-name $line --region ${account.region};
      done`,
    ] : []),
  ],
};

// tslint:disable-next-line: no-unused-expression
new PipelineApp(pipelineAppProps);

As you can see this is a very large stack with many input properties. In the future I might break down this huge stack into smaller ones. This example is interesting because it does a lot of testing with Postman and the AWS CLI at the end. The property testCommands performs tests for the DEV Stage Postman:

npx newman run test/alf-cdk.postman_collection.json --env-var baseUrl=$RestApiEndPoint -r cli,json --reporter-json-export tmp/newman/report.json --export-environment tmp/newman/env-vars.json --export-globals tmp/newman/global-vars.json`

Then it deletes any remaining CDK stacks resulting from the tests:

aws cloudformation describe-stacks --query "Stacks[?Tags[?Key == 'alfInstanceId'][]].StackName" --region ${account.region} --output text |
awk '{print $1}' |
while read line;
  do aws cloudformation delete-stack --stack-name $line --region ${account.region};
done`,

Summary

CDK code pipelines are super cool. But they are also super complicated or complex. You don't want to deal with all this complexity every time just to squeeze stacks into a staging pipeline. So there should be a good abstraction to these staging pipelines.

In this article I presented such an abstraction in form of a library. I am already using it to run some CDK stacks, which are presented here.

This library can only access repositories in my github account. It would be possible to make the library more general sophisticated in order to be able to access any git repositories. If you find this useful, let me know. But now I'm curious. What do you think of my PipelineApp library? What can I improve? Let me know.

Become a Patreon!

AWS CDK Let's build a Platform - Api Gateway

Sun, 13 Sep 2020 00:00:00 GMT

Hi CDK Fans,

Welcome back to my little series "AWS CDK Let's build a platform". In the previous episode AWS CDK Let's build a Platform - Frontend I explained how we build a cool static React Web App with AWS CDK for my company unimed.de.

AWS CDK is a framework for creating and using Cloudformation templates. You can choose between common programming languages like Java, Python or TypeScript. We have decided to use TypeScript. If you want to know more about AWS CDK I recommend my other posts here in my blog like cdk-example.

In this episode I will talk about the AWS Api Gateway which is used by the frontend application to retrieve or store internal data.

Api Gateway

The AWS Api Gateway is the heart of many AWS deployments. It is a rest api for processing requests and the following responses. For example, if internal data is to be requested, this could be done with this request:

GET https://example.com/users

The response body could then look like this:

[
    {"user": "Alice", "age":18},
    {"user": "Bob", "age":19}
]

So the Api Gateway receives a request, forwards it to an integration and then returns a response. The integration can be chosen from different technologies like Mock, HTTP Proxy, Lambda Proxy or other AWS services. When creating the API gateway, we decided to use the variant with the openApi specification. I already wrote about this in a previous post and recommend you to read it.

In short, using an openApi specification offers some advantages like schema validation of the request parameters and a great documentation about the expected responses and possible error messages. Especially when using a NoSQL database like DynamoDB the validation of the request parameters is extremely important because the database itself does not provide a schema! And this is how an openApi Api Gateway looks like in CDK:

const api = new SpecRestApi(this, this.stackName, {
    apiDefinition: ApiDefinition.fromAsset(join(__dirname, './path/to/api-gw.yaml')),
});

The openApi specification here called api-gw.yaml could look like this:

openapi: "3.0.1
info:
  title: "example-endpoints"
  version: "2020-09-01T13:03:56Z"
paths:
  /enumerations:
    get:
      security:
      x-amazon-apigateway integration:
        type: "aws_proxy"
        uri: "arn:aws:apigateway:${AWS::region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::region}:${AWS::AccountId}:function:example/invocations"
        passthroughBehavior: "when_no_match
        httpMethod: "POST
        credentials: "arn:aws:iam::@@ACCOUNT_ID@@@:role/apiRole"
    options:
      responses:
        200:
          description: "200 responses"
    ...

If you use Lambdas for your integrations you can also allow the Api Gateway to access your Lambdas right here:

const apiRole = new Role(this, 'apiRole', {
    assumedBy: new ServicePrincipal('apigateway.amazonaws.com'),
});

apiRole.addToPolicy(new PolicyStatement({
    resources: ['*'],
    actions: ['lambda:InvokeFunction'],
}));

Alternatively, the Lambdas themselves could give the Api Gateway permission to call them.

Part of our staging pipeline, which I will explain in one of the following parts, are integration tests that are executed after each commit. At the moment it is "only" postmant tests that are executed via Newman:

npx newman run test/postman/example-com.postman_collection.json --env-var baseUrl=http://example.com --env-var keycloakUrl=http://keycloak-dev.example.com --env-var user=${testUserName} --env-var pass=${testUserPassword} -r cli,json --reporter-json-export tmp/newman/report.json

The user credentials are previously set in the build via secure environment variables, which are loaded from AWS SecurityManager.

Summary

In this part of the "AWS CDK Let's build a Platform" series I have explained how the backend stack could look rough. What I left out on purpose is the information what comes behind the Api Gateway or Lambdas. Usually you access data from memory like maybe DynamoDB or RDS. But you might also want to trigger more complex processing with AWS Step Functions.

A great example how to do a CRUD implementation with API Gateway, Lambda and DynamoDB is described in another post (https://martinmueller.dev/cdk-example). Did you like the next part of my "AWS CDK Let's build a Platform" series? What do you want to know next? Let me know.

Become a Patreon!

AWS CDK Let's build a Platform - Frontend

Sun, 06 Sep 2020 00:00:00 GMT

Hi CDK fans,

Since I have the great opportunity to help building a platform for my company unimed.de, I want to explain how it looks like. At the moment we are working on an exciting platform for efficient storage and retrieval of internal data. My main responsibility is the AWS infrastructure.

Our DevOps team wants to leverage as much as possible to AWS. We use AWS CDK to manage the resources in AWS. AWS CDK is a framework for creating and using Cloudformation Templates. You can choose between common programming languages. We decided to use TypeScript because our frontend is also written in TypeScript and offers the advantage of strong type definition without loosing much flexibility. If you want to know more about AWS CDK I recommend my other posts here in my blog like cdk-example.

The above mentioned frontend is implemented in TypeScript flavour with React. I'm planning a multi-part series about "AWS CDK Let's build a Platform" and this episode is specifically about the frontend.

React Frontend

As already mentioned our frontend is a React Browser App in TypeScript flavor. It uses the material design wherever possible. Authentication is done via Keycloak which is connected to our internal Active Directory. After entering the access data, relevant data can be searched for and new data can be added. The static app is built with npm run build in the build folder.

At this point I would like to make a cut and tell more about the used infrastructure. The React Browser App is a Static Web App and needs a couple of AWS resources. I will list these in the next section and how they can be managed using CDK.

CDK Stack

To display the static React App an S3 Bucket is required, which serves as a static Web App Bucket:

const bucket = new AutoDeleteBucket(this, props.domainName, {
  bucketName: `${props.subDomain}.${props.domainName}`,
  websiteIndexDocument: 'index.html',
  websiteErrorDocument: 'index.html',
  removalPolicy: core.RemovalPolicy.DESTROY,
});

I use a CDK highlevel construct called AutoDeleteBucket which can delete itself if necessary. The normal S3Bucket Construct can only delete the bucket if there is no data in it. So the AutDeleteBucket first deletes all data contained and then removes itself. This flexible behavior is useful for buckets that are only used as static Web App containers. The name of the static Web App Bucket bucketName: ${props.subDomain}.${props.domainName} is usually given after the domain e.g. example.com .

The S3 bucket stores the static build and is connected to Cloudfront:

const cloudFrontOAI = new OriginAccessIdentity(this, 'OAI', {
  comment: `OAI for ${props.domainName} website.`,
});

const cloudFrontDistProps: CloudFrontWebDistributionProps = {
  aliasConfiguration: {
      acmCertRef: props.acmCertRef,
      names: [ `${props.subDomain}.${props.domainName}` ],
      sslMethod: SSLMethod.SNI,
      securityPolicy: SecurityPolicyProtocol.TLS_V1_1_2016,
  },
  originConfigs: [
    {
      s3OriginSource: {
        s3BucketSource: bucket,
        originAccessIdentity: cloudFrontOAI,
      },
      behaviors: [{ isDefaultBehavior: true }],
    },
  ],
  errorConfigurations: [
    {
      errorCode: 404,
      errorCachingMinTtl: 60,
      responseCode: 200,
      responsePagePath: "/index.html",
    }
  ]
};

const cloudfrontDistribution = new CloudFrontWebDistribution(
  this,
  `${props.subDomain}.${props.domainName}-cfd`,
  cloudFrontDistProps
);

Cloudfront is a Cloud Distribution Network (CDN) from AWS that cascades static Web Apps all over the world in the respective regions. This keeps latency during loading to a minimum.

To load the static Web App Build into the S3 Bucket you have to create a BucketDeployment Construct:

new BucketDeployment(this, 'DeployApp-${new Date().toString()}', {
  sources: [Source.asset("../build")],
  destinationBucket: bucket,
  distribution: cloudfront distribution,
  distributionPaths: ['/'],
});

The path for invalidating the Cloudfront distribution is also specified here in the BucketDeployment. This then ensures that the cash is reloaded if a new app is deployed. The static Web App build folder contains the build of the React App and was previously created with npm run build.

Furthermore a Route 53 Record Resource is required to point a custom domain (e.g. www.example.com) to the cloudfront endpoint to make the static web app accessible via URL:

const zone = HostedZone.fromLookup(this, 'Zone', { domainName: props.domainName });
new ARecord(this, 'SiteAliasRecord', {
  recordName: `${props.subDomain}.${props.domainName}`,
  target: AddressRecordTarget.fromAlias(new CloudFrontTarget(cloudfrontDistribution)),
  zone
});

And here is the whole code:

import { StackProps, Construct } from '@aws-cdk/core';
import { AutoDeleteBucket } from '@mobileposse/auto-delete-bucket'.
import { BucketDeployment, Source } from '@aws-cdk/aws-s3-deployment';
import {
  CloudFrontWebDistribution,
  CloudFrontWebDistributionProps,
  OriginAccessIdentity,
  SSLMethod,
  SecurityPolicyProtocol
} from '@aws-cdk/aws-cloudfront';
import { ARecord, AddressRecordTarget, HostedZone } from '@aws-cdk/aws-route53';
import { CloudFrontTarget } from '@aws-cdk/aws-route53-targets';
// @ts-ignore
import codedeploy = require('@aws-cdk/aws-codedeploy');
// @ts-ignore
import lambda = require('@aws-cdk/aws-lambda');
// @ts-ignore
import core = require('@aws-cdk/core');


export interface FrontendStackProps extends StackProps {
  stage: string;
  acmCertRef: string;
  domainName: string;
  subDomain: string;
}

export class FrontendStack extends core.stack {

  constructor(scope: Construct, id: string, props: FrontendStackProps) {
    super(scope, id, props);

    const bucket = new AutoDeleteBucket(this, props.domainName, {
      bucketName: `${props.subDomain}.${props.domainName}`,
      websiteIndexDocument: 'index.html',
      websiteErrorDocument: 'index.html',
      removalPolicy: core.RemovalPolicy.DESTROY,
    });

    const cloudFrontOAI = new OriginAccessIdentity(this, 'OAI', {
      comment: `OAI for ${props.domainName} website.`,
    });

    const cloudFrontDistProps: CloudFrontWebDistributionProps = {
      aliasConfiguration: {
          acmCertRef: props.acmCertRef,
          names: [ `${props.subDomain}.${props.domainName}` ],
          sslMethod: SSLMethod.SNI,
          securityPolicy: SecurityPolicyProtocol.TLS_V1_1_2016,
      },
      originConfigs: [
        {
          s3OriginSource: {
            s3BucketSource: bucket,
            originAccessIdentity: cloudFrontOAI,
          },
          behaviors: [{ isDefaultBehavior: true }],
        },
      ],
      errorConfigurations: [
        {
          errorCode: 404,
          errorCachingMinTtl: 60,
          responseCode: 200,
          responsePagePath: "/index.html",
        }
      ]
    };

    const cloudfrontDistribution = new CloudFrontWebDistribution(
      this,
      `${props.subDomain}.${props.domainName}-cfd`,
      cloudFrontDistProps
    );

    new BucketDeployment(this, 'DeployApp-${new Date().toString()}', {
      sources: [Source.asset("../build")],
      destinationBucket: bucket,
      distribution: cloudfront distribution,
      distributionPaths: ['/'],
    });

    const zone = HostedZone.fromLookup(this, 'Zone', { domainName: props.domainName });
    new ARecord(this, 'SiteAliasRecord', {
      recordName: `${props.subDomain}.${props.domainName}`,
      target: AddressRecordTarget.fromAlias(new CloudFrontTarget(cloudfrontDistribution)),
      zone
    });
  }
}

Pipeline

We deploy the static React app via a staging pipeline. This means the app runs through the different stages Dev, QA and Prod. All stages are separate accounts. This way we achieve increased security by separating the resources. The dev account is used to test new features. The QA environment serves as the next stage.

I would like to report in more detail about our Staging Pipeline in the next part of the series.

Summary

Building a platform using AWS is exciting and fun. Having the complete infrastructure as code is extremely beneficial. It allows a sophisticated staging from e.g. a Dev environment to QA and then to Prod. Furthermore the code serves as documentation about what exactly has been and will be deployed in AWS. It is also much less error-prone than having to click everything together manually.

Here I presented the first part of my "AWS CDK Let's build a Platform" series. I explain which resources our static Web App needs in AWS and how they are managed with CDK. In the next part I will talk in more detail about our staging pipeline which is reused almost everywhere to get CDK apps through the different stages Dev, QA and Prod. I hope you liked the article and if you have any questions, just write me.

Become a Patreon!

Alfresco Signatures with Sinadura

Fri, 14 Aug 2020 00:00:00 GMT

Sinadura Addon

Sinadura.net is an open source project for digital signature software. For this purpose a Sinadura server is normally running for signature check and creation. A client then connects to the Sinadura server.

Sinadura has an Alfresco addon repo in GitHub https://github.com/zylklab/alfresco-sinadura which only works with ACS Community and Enterprise version 5. Of course it would be great to support newer Alfresco versions with Docker. So I made it my business to make this great addon compatible with ACS 6.2.

ACS 6.2 Compatible

Currently the Alfresco Sinadura addon in https://github.com/zylklab/alfresco-sinadura is still based on SDK 2.2.0 and ACS 5. It seems that SDK 2.2.0 does not work ootb. anymore and some necessary artifacts are not available in the Alfresco Nexus. Only with a lot of efforts would it be possible to compile the Sinadura amps with SDK 2.2.0. So I decided to use the latest SDK version 4.1.0.

The migration, which you can find in my Git Repo at https://github.com/mmuller88/alfresco-sinadura-6X-pmgatech, worked great. How to build an Alfresco SDK 4.1.0 build is very well described in the official Alfresco SDK Repo. SDK 4.1.0 is heavily based on Docker and uses Docker Compose as a container orchestrator.

A bit more challenging was the migration of the AMP structure to 4.1.0 as some files have to be placed in a different location than in 2.2.0. Also I had to put the sinaduraCloud war file in its own Tomcat running with Java 8 as Alfresco uses Java 11.

Summary

Digital signatures are an exciting topic and enable an Alfresco document management system to have completely new use cases. The process of signing in conjunction with Alfresco Process Service could also be interesting.

Many thanks to my sponsor www.pmga.tech for the migration work on Sinadura. Do you have exciting Alfresco projects you are working on? Let me know :) !

Become a Patreon!

Create ACS Infrastructure easy with AWS CDK

Sun, 26 Jul 2020 00:00:00 GMT

Hi Alfrescans.

Infrastructure as code has been popular in DevOps for quite some time. One of the first pioneers of IaC (Infrastructure as Code) is AWS with Cloudformation. For this, cloudformation templates must be written in JSON or YAML. This works very well but is a time consuming and complex work.

With AWS CDK, a new level of abstraction has been created above the cloudformation templates, which makes it easier than ever to provide the infrastructure for a running ACS. In the next sections I will explain what AWS CDK actually is and how it can help to build AWS infrastructure for ACS.

What is AWS CDK?

CDK stands for Cloud Development Kit and is an open software development framework for modeling and managing Cloudformation resources and thus AWS infrastructure. AWS CDK synthesizes to Cloudformation Templates. This means that Cloudformation Templates are being created from the CDK code. A so-called CDK App consists of one or more Cloudformation Stacks. It supports well-known languages like Java, Python, JavaScript or TypeScript. For the rest of this article I use TypeScript.

Why is CDK so great?

The great thing about CDK in the TypeScript flavor are the type definitions which allow for fast progress when developing with an IDEA, since it virtually brings documentation with it thanks to the types. Also, the years of experience with cloud formation have already resulted in excellent defaults, which rarely need to be overwritten. CDK also takes care of the upload of the synthesized cloudformation template and assets into an S3. Also, the CDK framework is open source and if you want to take a closer look at how CDK works, you can do so in GitHub.

So AWS CDK is an excellent way to provide the infrastructure for an ACS deployment. In the following section, I'll explain the two stacks Ec2 stack and EKS stack created with CDK.

CDK Constructs

Constructs can be either low level or high level. Low level constructs represent a Cloudformation resource. High level constructs are a combination of several low level constructs. The whole CDK app represents a tree with the app itself as root. After that come the high level constructs and the leaves are the low level constructs. For completeness, I should add that low level constructs are getting auto generated from existing Cloudformation resources.

A good example is the high level construct Vpc from the package @aws-cdk/aws-ec2 which is a combination of resources like Vpc, InternetGateway, Cidr or VpC Iam Role. The creation of the high level VPC Construct looks like this:

import { Vpc } from '@aws-cdk/aws-ec2';
import { Stack, App} from '@aws-cdk/aws-ec2';

class VpcStack extends Stack {
  constructor(app: app, id: string) {
    super(app, id);

    new Vpc(this, 'vpc'); // just this line
  }
}

const app = new App();
new VpcStack(app, 'VpcStack');

app.synth();

Only the line new Vpc(this, 'vpc'); is required to define a complete public VPC. Doing the same in a Cloudformation template would be 200 lines long! This is made possible because CDK has extremely useful default settings. You can see how well this framework benefits from years of experience with Cloudformation. If you would like to create a private VPC you only need to pass a third parameter object:

new Vpc(this, 'CustomVPC', {
  cidr: '10.0.0.0/16',
  maxAzs: 2,
  subnetConfiguration: [{
    cidrMask: 26,
    name: 'isolatedSubnet',
    subnetType: SubnetType.ISOLATED,
  }],
  natGateways: 0
});

Ec2 Stack

The code is public and in my GitHub Repo. The ACS deployment was generated using the Alfresco Docker Installer and uses Docker Compose to orchestrate the ACS deployment services. The CDK app is defined in the file index.ts. If no CDK Apps have been deployed yet, the CDK Toolkit Stack must be created in the target region with:

cdk bootstrap aws://<ACCOUNT-NUMBER>/<REGION>

Then the stack is deployed with the following command:

npm run build && cdk deploy

If everything went well, the Alfresco Content App should be available on the public DNS Url for a short ACA. For example, ec2-34-201-46-76.compute-1.amazonaws.com. You can get that URL directly from the AWS browser in the Ec2 section or define an Output resource and your CDK stack.

The CDK app creates a cloudformation stack that deploys an Ec2 instance behind an application load balancer. The ACS Deploy Code, which is also contained in the same repo, is copied to the Ec2 instance and ACS is started. Docker Compose serves as a container orchestrator.

This is ideal for simple developer environments, as Docker Compose templates can be created quickly and are easily extensible. However, if you want to create a production-like ACS deployment, you should use Kubernetes as a container orchestrator.

EKS Stack

EKS stands for Elastic Kubernetes Service and provides a Kubernetes Masternode when used. The Ec2 worker nodes are also configured. In contrast to the Ec2 stack, the EKS stack uses Kubernetes to orchestrate the ACS deployment services. There are already some good examples how to create an EKS deployment with CDK. Here you can find my GitHub code. The following properties are needed to create the cluster:

const eksCluster = new Cluster(this, 'cluster', {
  clusterName: this.stackName,
  mastersRole: eksClusterAdmin,
  vpc: vpc,
  kubectlEnabled: true, // we want to be able to manage k8s resources using CDK
  defaultCapacity: 0, // we want to manage capacity our selves
  version: CubernetesVersion.V1_17,
});

I overwrite the cluster name generated by the CDK with clusterName because it is not reader-friendly. The masterRole allows my IAM User Admin access to the cluster. The workernode is then defined as follows:

const onDemandASG = new AutoScalingGroup(this, 'OnDemandASG', {
  vpc: vpc,
  role: workerRole,
  minCapacity: 1,
  maxCapacity: 1,
  instanceType: new InstanceType('t2.large'),
  machineImage: new EksOptimizedImage({
    cybernetic version: '1.17',
    nodeType: NodeType.STANDARD // without this, incorrect SSM parameter for AMI is resolved
  }),
  updateType: UpdateType.ROLLING_UPDATE
});

To make Alfresco accessible via URLs, you need to define the address as well. I use a Nginx Ingress in AWS Flavour:

new HelmChart(this, 'NginxIngress', {
  cluster: eksCluster,
  chart: 'nginx-ingress',
  repository: 'https://helm.nginx.com/stable',
  namespace: 'kube-system',
  values: {
    'rbac': {
      'create': true
    },
    'controller': {
      'scope': {
        'enabled': true,
        namespace: acsNamespace
      },
      'config': {
        'force-ssl-redirect': true,
        server tokens: false
      },
      'service': {
        'targetPorts': {
          'https': 80
        },
        annotations: {
          'service.beta.kubernetes.io/aws-load-balancer-backend-protocol': 'http',
          'service.beta.kubernetes.io/aws-load-balancer-ssl-ports': 'https',
          'service.beta.kubernetes.io/aws-load-balancer-ssl-cert': awsCertArn,
          'external-dns.alpha.kubernetes.io/hostname': `${acsNamespace}.eks.alfpro.net`
          'service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy': awsCertPolicy
        },
        publishService: {
          'enabled': true
        }
      }
    }
  }
})

After successful deployment with

npm run build && cdk deploy

the following output appears:

 ✅ AcsEksCluster

Outputs:
AcsEksCluster.ClusterConfigCommand43AAE40F = aws eks update-kubeconfig --name AcsEksCluster --region us-east-1 --role-arn arn:aws:iam::1111122223333:role/AcsEksCluster-eksClusterAdminE955DB57-1H9KJVEE241KS
AcsEksCluster.clusterGetTokenCommand06AE992E = aws eks get-token --cluster-name AcsEksCluster --region us-east-1 --role-arn arn:aws:iam::1111122223333:role/AcsEksCluster-eksClusterAdminE955DB57-1H9KJVEE241KS

The aws eks update-config command can be used to connect to the cluster and apply cluster configurations using kubectl.

Helm 3 Issue

Unfortunately none of the ACS Charts, i.e. Community and Enterprise, are compatible with Helm 3 at this time. This means we can not use the existing HelmChart CDK Constructor to install ACS. Alternatively, ACS must be installed with Helm 2.

Summary

AWS CDK is an exciting framework for building Alfresco infrastructure in AWS. CDK synthesizes into cloud formation templates, so there is little or no need to change existing CI CD pipelines. The gain from using CDK is huge. E.g. the type definition prevents bugs already in the editor. Many lines of code are saved and there is no or hardly any need to template YAML. For me CDK is the next logical step for IaC in AWS. There are even the first efforts to integrate CDK into the Terraform ecosystem, which I find very exciting. I would be really interested to hear of you experiences with CDK :)!

Become a Patreon!

AWS CDK Apps deployed with Lambda

Sat, 11 Jul 2020 00:00:00 GMT

Ahoy AWS CDK Fans

Creating any number of Ec2 instances with a Lambda is no problem. But if you want to create more complex deployments such as LoadBalancer, Securitygroups, Scalingroups, etc. with a Lambda that is tricky! CDK Apps are a good choice to manage those stacks. Deploying CDK Apps with a Lambda is not that easy and there is less information about it in the web.

During a CDK Meetup I got the suggestion to use a CodeBuild project to create the CDK App via Lambda. And yes that worked out great. In the next sections I will explain why this is useful for me and how I did it.

Use Case

I already mentioned several times in my blogs that I use CDK to deploy my Alfresco Provisioner(also: CDK Construct Solutions). Running Alfresco only on an EC2 VM works, but is not the best solution in long terms. I also want to be able to run a load balancer and multiple EC2 instances for an Alfresco stack. Even more I would like to integrate a Kubernetes cluster in the stack to orchestrate the instances and pods. So it was necessary to manage this kind of complex deployments and CDK Apps are perfect for that.

Another use case is for a customer who wants to be able to deploy CDK stacks in different stage accounts like Dev, QA, Prod. The deployment should be supervised by CodePipeline written in CDK. This use case differs in that the stack is not deployed via Lambda, but via CodePipeline and CodeBuild. For deploying CDK stacks using CodePipeline a CodeBuild project must be created.

CDK Deploy via CodeBuild and Lambda

To deploy stacks using CDK, a CodeBuild project must be created. The project looks like this:

const createInstanceBuild = new Project(scope, 'LambdaBuild', {
    role: createInstanceBuildRole, // role needs all permission for deploying Stacks, accessing S3, logs ...
    source: gitHubSource, 
    buildSpec: codebuild.BuildSpec.fromObject({
    version: '0.2',
    phases: {
        install: {
            commands: [
                'cd src',
                'npm install -g aws-cdk',
                'npm install',
            ],
        },
        build: {
            commands: [
                'npm run build',
                'cdk deploy --require-approval never'
            ]
        },
    },
    }),
    environment: {
        buildImage: codebuild.LinuxBuildImage.STANDARD_4_0,
    },
});

And it's that easy! CDK Apps with CodeBuild can be deployed within an account. But if you want to do cross deployment, you only have to adjust the commands section slightly:

...
phases: {
        install: {
            commands: [
                `aws --profile dev configure set aws_access_key_id $AWS_ACCESS_KEY_ID_DEV`,
                `aws --profile dev configure set aws_secret_access_key $AWS_ACCESS_KEY_ID_DEV`,
                `aws --profile dev configure set region $AWS_ACCESS_KEY_ID_DEV`,
                `aws --profile prod configure set aws_access_key_id $AWS_ACCESS_KEY_ID_PROD`,
                `aws --profile prod configure set aws_secret_access_key $AWS_ACCESS_KEY_ID_PROD`,
                `aws --profile prod configure set region $AWS_ACCESS_KEY_ID_PROD`,
                'cd src',
                'npm install -g aws-cdk',
                'npm install',
            ],
        },
        build: {
            commands: [
                'npm run build',
                'cdk deploy --require-approval never --profile dev'
            ]
        },
    },
}),

If you would like to take a look at the required stack changes beforehand, you can easily do that with:

cdk diff --profile dev

What's missing now is the Lambda implementation to run the CodeBuild project. First the Lambda must be created:

const createInstanceLambdaRole = new Role(scope, 'createInstanceLambdaRole', {
      assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
      managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole')],
    });

    createInstanceLambdaRole.addToPolicy(new PolicyStatement({
      resources: ['*'],
      actions: ['codebuild:StartBuild', 'logs:*']
    }));

new Function(scope, 'createCdkApp', {
    code: new AssetCode('lambda'),
    handler: 'create-cdk-stack.handler',
    runtime: Runtime.NODEJS_12_X,
    environment: {
        PROJECT_NAME: createInstanceBuild.projectName
    },
    role: createInstanceLambdaRole
});

As the Lambda CDK code already indicates, the Lambda function is located in the folder ./lambda:

// lambeda/create-cdk-stack.ts
import AWS = require("aws-sdk");
import { CodeBuild } from "aws-sdk";

var codebuild = new AWS.CodeBuild();

const PROJECT_NAME = process.env.PROJECT_NAME || ''

export const handler = async (event: any = {}): Promise<any> => {
  console.debug("create-cdk-stack event: " + JSON.stringify(event));

  const params: CodeBuild.Types.StartBuildInput = {
    projectName: PROJECT_NAME
  };
  console.debug("params: " + JSON.stringify(params));
  const startBuildResult = await codebuild.startBuild(params).promise();
  console.debug("startBuildResult: " + JSON.stringify(startBuildResult));
}

E voila and can independently deploy the Lambda CDK Apps.

Summary

CDK makes stack creation and management easy. This is also useful in dynamically created (e.g. with a Lambda) stacks! With previous solutions, cloudformation templates had to be created first and then uploaded to an S3 which could then be used as the source for a cloudformation deployment.

With this new approach I presented that is no longer necessary and you can concentrate more on the creation of the CDK app itself. Another advantage I appreciate is that I can test the CDK apps deployed via Lambda separately without having to deploy the complete CDK Parent Stack. This combined with CI CD Pipeline is the DevOps dream comes true!

Become a Patreon!

Optimise Infrastructure with CDK Solutions Constructs Part II.

Sat, 04 Jul 2020 00:00:00 GMT

Hi CDK Fans

In the previous Part I I have built the DynamoDB Stream to Lambda Solution Construct into my Alfresco Provisioner CDK Deployment and discussed the benefits of doing so. This week I integrated more CDK Solutions Constructs. CDK Solutions Constructs are CDK Constructs which encapsulate often used Cloudformation Patterns. You can read more about them in Part I.

Cloudfront S3

In GitHub, this is the Cloudfront S3 Solution Construct. It implements a Cloudfront in front of an S3 bucket. This allows for a higher performance distribution of files because they are cached and distributed on the edge using a content delivery network. With the Cloudfront S3 Solution Pattern comes some great well-architected aspects like HTTP security headers and buckets for CDN and S3 access logs.

const { CloudFrontToS3 } = require('@aws-solutions-constructs/aws-cloudfront-s3');

new CloudFrontToS3(stack, 'test-cloudfront-s3', {
    deployBucket: true
});

I planed to use this pattern to extend the S3 Static website, because it already has a cloudfront as well. However, the Cloudfront S3 Solution Construct does not support static websites. Therefore I cannot use it for my stack. What matters is that I learned a lot about this solution construct and now I know what I can't use it for, which is useful!

SQS Lambda

The SQS Lambda Solution Construct is an exciting pattern that connects a queue from the Simple Queue Service (SQS for short) to a Lambda and thus forwards the queue messages to the Lambda for further processing. The queue and the Lambda only receive the required permissions. In addition, the SQS Lambda Solution Construct also deploys a Dead Letter Queue that is used to store and process failed messages.

const { SqsToLambda } = require('@aws-solutions-constructs/aws-sqs-lambda');

new SqsToLambda(stack, 'SqsToLambdaPattern', {
    deployLambda: true,
    lambdaFunctionProps: {
        runtime: lambda.Runtime.NODEJS_12_X
        handler: 'index.handler',
        code: lambda.code.asset(`${__dirname}/lambda`)
    }
});

I need this Solution Construct because the DynamoDB Stream to Lambda Solution Construct mentioned in Part I showed a strange behaviour by occasionally duplicating event messages. With the SQS Lambda Solution Construct I was able to create a FIFO SQS which now has the additional advantages mentioned in the previous paragraph.

Lambda DynamoDB

A simple but nevertheless great Solution Construct is the Lambda DynamoDB. It perfectly standardizes the connection of a Lambda to a DynamoDB. If required, an existing DynamoDB table can be used or a new one created:

const { LambdaToDynamoDB } = require('@aws-solutions-constructs/aws-lambda-dynamodb');

new LambdaToDynamoDB(stack, 'test-lambda-dynamodb-stack', {
    deployLambda: true,
    lambdaFunctionProps: {
        code: lambda.code.asset(`${__dirname}/lambda`),
        runtime: lambda.Runtime.NODEJS_12_X,
        handler: 'index.handler'
    },
    //optionally pass existing table
    existingTableObj: ddbTable
});

As part of the solution construct the DynamoDB access privileges are granted to the Lambda. Also, after the table is created or forwarded, an environment variable is passed to the Lambda, which contains the name of the table. This way the table is easy accessible with the Lambda:

// Copied from Solution Construct Repo
this.lambdaFunction.addEnvironment('DDB_TABLE_NAME', this.dynamoTable.tableName);

// Usage within the Lambda
const DDB_TABLE_NAME = process.env.DDB_TABLE_NAME || ''

Summary

The next CDK Solutions Constructs could be successfully integrated. When working with the CDK Solution Constructs I ask myself if it makes sense to build own Solution Constructs. This would have the advantage of abstracting the infrastructure. However, the disadvantages are that not much consideration is given to the well-architected Framework.

A further disadvantage would be that the self-created Solution Constructs should of course be maintained and further developed, which would require an additional efforts. The Use Case will decide about the sense of creating your own Solutions Constructs. Until then, you can and should use the existing AWS CDK Solution Constructs, because they are open source on Git and the strong AWS community maintains and develops them further. I could even contribute back to fix a bug. Let me know about your experience with CDK Solutions Constructs :).

Become a Patreon!

Optimize Infrastructure with CDK Solutions Constructs Part I.

Sun, 28 Jun 2020 00:00:00 GMT

Ahoy AWS'ler

Recently the first CDK Solution Constructs were released. It promise a new level of abstraction for CDK Constructs by combining often used Cloudformation patterns into their own CDK Constructs. For example, if you want to use an API GateWay with a lambda proxy, you can use the aws-apigateway-lambda Solution Construct.

A great AWS Blogpost demonstrates the combination of two such Solution Constructs very well. Furthermore, the CDK Solution Constructs promise to align on the well-architected Framework, in order to include the five pillars Operational Excellence, Security, Reliability, Performance Efficiency and Cost Optimization best as possible in the CDK deployment. This and the reduction in CDK code lines has led me to investigate whether my Alfresco Provisioner CDK Deployment could benefit from some CDK Solution Constructs. In the next few sections I will describe the CDK Solution Pattern that I have successfully incorporated into my deployment so far.

AWS DynamoDB Stream to Lambda

This pattern, also to be seen in Github, describes the often used intention to call a DynamoDB table with changes in the items via a stream to a Lambda in order to be able to react to changes like creating, editing or deleting of the item itself. An example could be if the item is deleted from the table, it can be persisted elsewhere at a lower cost.

In my deployment I use it directly to map the EC2 instances according to the desired configuration in the table. For example, if the EC2 instance needs to be stopped, only stopped is written to the expectedStatus column of the table. If the instance is to be running again you simply write running. If the instance is to be completely terminated, the item can simply be deleted and the streamed lambda will ensure that the instance is deleted.

My previous solution was to use StepFunction alone, but using streams is much more advantageous and saves many lines of code. I also like the fact that the pattern abstracts the stream away and I only have to worry about the table and the lambda.

Summary

The CDK Solution Constructs are great patterns that simplify the creation of the AWS infrastructure massively and even take the well-architected framework into account. Next week I will try to integrate the next pattern Cloudfront S3. Be curious!

With great excitement I follow the development of the CDK Solution Constructs. Do you already have experience with CDK or even with the CDK Solution Constructs? If so for what? If you have any questions or suggestions, let me know!

Become a Patreon!

ACS and APS Integration

Sun, 14 Jun 2020 00:00:00 GMT

Hi Alfrescans.

Some months ago I reported about an exciting project with the customer where we used the latest ACS and APS version. So far everything has been running smoothly. But now we were facing an exciting challenge. We wanted to start processes in APS from ACS. But without using the Share Connector, which normally facilitates such integrations between ACS and APS. Since ACS 6.0 the Share Connector is no longer supported by Alfresco.

Problem

ACS and APS are two independent services that are able to communicate with each other via CMIS or REST APIs. For a long time it has been possible to set ACS as a content management system and thus store and retrieve data directly in ACS (content and properties). This works great as long as the processes are started from APS. But you can also perfectly imagine the use case that processes should be started from ACS. An example would be if a document is uploaded in a certain ACS folder, a process in APS should be started with this document.

For ACS 5.2 this problem could be solved with the Share Connector. This extends ACS with webscripts which can start processes via REST with APS. Additionally the Share Connector offers some Share UI extensions for the APS integration. It could be still possible to use the Share Connector for newer ACS versions, but then Alfresco would no longer offer support if you are running in any problems with it. In the following I explain two methods how to start a process in APS from a URL without using the Share Connector. Then I explain how to call this URL in ACS as part of an action webhook.

Custom Endpoint for Starting a Process

APS offers the possibility to implement custom endpoints via Java code. This can then be used to start processes and transfer variables to the process. You can find the Alfresco documentation here. If such a custom endpoint is implemented, it can be used as a WebHook in ACS to start a process from ACS. As described in the documentation, the endpoint is defined using Java and then packed into a jar. The configuration of this was a bit complex because some dependencies are only available in the private Alfresco Nexus. I needed the following dependencies to build the JAR:

<dependencies>
    <dependency>
        <groupId>org.alfresco</groupId>
        <artifactId>alfresco repository
    </dependency>
    <dependency>
        <groupId>com.activiti</groupId>
        <artifactId>activiti-app-rest</artifactId>
        <version>1.11.0</version>
    </dependency>
    <dependency>
    <groupId>org.springframework</groupId>
        <artifactId>spring-web>/artifactId>
        <version>4.1.6.RELEASE</version>
    </dependency>
</dependencies>
...
<repositories>
    <repository>
        <id>enterprise-releases</id>
        <url>https://artifacts.alfresco.com/nexus/content/repositories/activiti-enterprise-releases</url>
    </repository>
</repositories>

Access to the private Alfresco Nexus repository is required. These can be obtained from support.alfresco.com . Afterwards the JAR can be copied into the container using a docker. Below I show the Docker Compose and Docker Settings for loading the JAR into the Tomcat webapp lib.

process:
    build:
        context: ./process
    environment:
    ...

The following Docker file is located in the ./process folder.

FROM alfresco/process-services:1.11.0

ARG TOMCAT_DIR=/usr/local/tomcat

COPY target/acsaps-1.0.0-SNAPSHOT.jar $TOMCAT_DIR/webapps/activiti-app/WEB-INF/lib

Better you test the custom endpoint with Postman before you try to call it with ACS Url Webhook. I'm not sure yet what the advantages of this method are compared to the APS Signals method in the next section. But I suspect that with Java programming a more complex process call might be possible.

APS Signal for Starting the Process

The use of start signals in APS is a different way of connecting ACS and APS. The big advantage compared to the custom endpoint method is that no Java code has to be created and everything can be configured via UI in /activiti-app. Another advantage is that the so called Start Signals can start several independent processes. In our example I will stick to one process. jtsmith describes the procedure very well in his blog post Start Signal Event with REST example. In short, a Basic Auth Endpoint is created using APS. Afterwards a signal process is modeled, which will intercept the signal and map it to the endpoint. It is important that the request mapping has the following form:

APS Signal Payload

{
   "signalName": "mysignal",
   "tenantId": "tenant_1",
   "async": "false",
   "variables":
   [
        {
            "name": "document",
            "value": "${document.nodeRef}"
        }
    ]
}

Please read the exact functions of the properties in the linked original post. Next you can create processes that can use the mysignal signal as start event to start the process. The model for the signal process can then be easily exported and reused.

ACS Webhook Action

The previous sections explained how to start an APS process from a URL. Now the only thing missing is the possibility to do exactly that with ACS. I consider the use case that a user wants to start a process when a file in a folder is uploaded to ACS. Two elements are necessary for this. For the folder we have to create a rule that is activated when a new document is created. Then a WebHook action must be executed.

Unfortunately Alfresco does not offer a Webhook action ootb. There are two possibilities to archive that. Either you create an Alfresco Webhook action yourself or use the sophisticated Webhook action from Acosix GmBH. That comes with a lot of great features like FreeMarker text input fields to define the payload or the Webhook URL, which is pretty cool. The Webhook Action works with payload templates that can be created / customized at runtime.

I have provided an APS Payload Template which already provides the APS Signal Payload mentioned above. If you want to transfer more variables like Alfresco properties using the Webhook, you only have to extend the payload template in the variables section. The Webhook Action from Axel also offers excellent debugging options to find possible bugs.

Summary

ACS and APS are powerful tools that combine to help companies finally find the long awaited, optimal digital solution. The integration of ACS with APS presents challenges for many Alfresco engineers. I've listed two ways in which this integration can be accomplished and hope it helps you achieve your ECM BPM goals using ACS and APS. Our ACS to APS integration at the customer site is far from complete as we plan to write complex processes in APS and are already thinking about the integrity between ACS and APS, which presents us with exciting problems. Write me how your ACS APS integration looks like :) .

Become a Patreon!

OpenApi generated Restic REST Server

Sat, 06 Jun 2020 00:00:00 GMT

Hello folks :).

In this article I want to tell you how easy it can be to create a backend REST server using OpenApi specifications. My ultimate goal is to make backups and the restore process of Alfresco data as easy as possible. After much research I decided to use Restic as my backup and restore engine. In my previous deployments all Alfresco data, i.e. blob data, DB data, Solr data etc. is stored in the /data folder. The automated backup works fine so far. Many thanks at this point lobaro and his brilliant restic backup work which helped me a lot in the part automated backup.

But now the restoring part is still to be done. And as it seems there is no tooling for it yet. So the administrator would have to do the restoring manually with restic. That means logging into the machine and executing the necessary Restic Restore commands, which restore the desired snapshot. I want to simplify this process and provide the administrator with a convenient UI for restoring. For this I use the OpenApi technologies. In the next sections I will tell you more about OpenApi and what exactly I did.

OpenApi

OpenApi which was formerly called Swagger is a YAML or JSON template language for describing RESTful APIs. In the following I describe what is great about OpenApi. First of all, the templates are extremely well suited as documentation about the API itself, because a nice looking HTML UI can be generated from the template, which describes the API endpoints very well. Such a UI can be seen on the cover of this blog post. Even more ingenious is the UI can be used directly to test the endpoints, i.e. to send and receive requests and responses. Many API interfaces, such as AWS API Gateway, offer to do the parameter validation of requests via OpenApi files. What is meant by parameter validation I will try to explain with the following example:

parameters:
    - in: query
    name: userId
    description: Get items of that user
    required: true
    type: string

Here you can see a parameter of type Query. This means it would look like this in the URL

http://<url>/items?userId=martin

With the parameter validation I can then define certain properties of the parameter, like here the name userId if it is required and what type the value should have, in our case as of type string.

Also very powerful is the property that it is possible to generate from OpenApi files client libraries and server stubs. For example, Alfresco generates JavaScript clients using the API Explorer (see GitHub Api-Explorer) and ADF (or ADF JS Github). There you can create a JavaScript API library from the Swagger File which can be used as a wrapper for the API requests in the ADF Components Github.

Also cool is, Postman offers an import function for OpenApi files. Then a collection is created from it. This is very handy if you want to start writing the requests in Postman.

Server generation

In this part I tell you more about my implementation in Github. The OpenApi Generator is a powerful tool for creating client libs and server stubs of OpenApi specifications like the one I have in repo ./restic.yaml . But at the moment I'm only interested in the server stub generation. It can be chosen from a variety of different server technologies like Go, Kotlin, Java, JavaScript. I have specifically chosen the NodeJS Express Server. To generate the server the following command is executed in ./build_server:

PWD=$(pwd)
docker run --rm -v ${PWD}:/local openapitools/openapi-generator-cli generate \
-i /local/restic.yaml \
-g nodejs-express-server \
-o local/server

An additional bonus. The NodeJS Express Server also generates a Swagger UI which is then accessible on /api-doc . Since I don't want to implement the handlers for the endpoints again and again when generating the server, I have placed them in a designated folder ./handlers . Then you only have to refer to it in the OpenApi Spec:

operationId: getSnapshots
description: List all snapshots
x-eov operation handler: handlers/default controller

The x-eov-operation-handler property refers to the handler function. The ./handlers folder then only needs to be copied to the Docker image.

Testing

For testing, I'm using the new GitHub build engine GitHub Actions to generate workflows. In my previous project, I tested GitHub Actions extensively with a Let's Encrypt SSL Docker Companion and found it to be good. In my test workflow under ./.github/workflows/action.yml the image is started with Docker Compose and tested extensively with the help of Postman. If you are more interested in automated testing with Postman just click here

Summary

Building REST servers is usually a complex process that requires writing a lot of code. Request parameters in the path or body must be validated for correctness. Or Url mapping algorithms must be equipped with request and response handler logic. And so that the user of the REST server does not despair, the REST API should be well documented. All these tasks and many more can be solved by the OpenApi approach.

With great works like the OpenApi Generator you can easily generate rest servers and document them with Swagger HTML UI. Also the REST server can be tested without any problems in the Swagger HTML UI with the try out function and postman tests can be generated directly from the OpenApi Spec File. Working with OpenApi specs is very exciting for me and I hope I could animate you to try OpenApi technologies. Let me hear how it went for you!

Become a Patreon!

Alfresco meets Let's Encrypt

Sat, 30 May 2020 00:00:00 GMT

Hi Alfrescans.

During the Alfresco Hackathon in May 2020 I implemented a Docker Companion extension for the Docker Alfresco Installer to manage SSL certificates. These certificates can be used for HTTPS connections. The SSL certificates are issued and regularly renewed by Let's Encrypt. Let's Encrypt also serves as the authorizer of the certificates. Pretty cool, right? So I don't have to worry about a secure, encrypted and maintained connection to my Alfresco proxy.

Unfortunately, the pull request was denied because the Alfresco Installer docker should remain in focus demo and trial. To make this feature more accessible for you, I decided to implement it into my GitHub repo and present it to you here. Additionally, I've written automated tests that use the new build engine GitHub Actions to test the Let's Encrypt Docker Companion. In the next sections I will explain the extension and the automated tests.

Docker Companion

The code for the Let's Encrypt extension is at me on GitHub. The Docker Compose deployment is started with the script ./start.sh. If you want Lets Encrypt to request an SSL certificate, you need a server where Docker Compose Deployment is running and a domain that redirects to this server. I have taken EC2 VM from AWS and created a CNAME record which points from my domain to the Public DNS name of the EC2 VM. The CNAME record then looks something like this:

a.notreal.net. CNAME ec2-3-8-139-83.eu-west-2.compute.amazonaws.com

The Alfresco Docker Compose Deployment can then be launched as follows:

./start.sh -spr https -sh a.notreal.net -sp 443

Thus the entire ACS Docker Compose Deployment is specifically switched to https and the domain a.notreal.net. The Docker Compose Deployment could also be started without using the ./start.sh script. However, the variables in the Docker Compose file would then have to be set correctly!

The SSL specific adjustments in the Docker Compose File only include the three following services:

    ...
    proxy:
      image: nginx:alpine
      depends_on:
        - alfresco
        - solr6
        - share
        - content app
      volumes:
        # redirect Alfresco Apps to port 80
        - ./config/nginx.conf:/etc/nginx/nginx.conf
      environment:
        VIRTUAL_PORT: 80
        VIRTUAL_HOST: ${SERVER_HOST}
        LETSENCRYPT_HOST: ${SERVER_HOST}
        LETSENCRYPT_EMAIL: admin@${SERVER_HOST}


    nginx-proxy:
      image: jwilder/nginx-proxy
      ports:
        - "80:80"
        - "443:443"
      volumes:
        - "/etc/nginx/vhost.d"
        - "/usr/share/nginx/html"
        - "/var/run/docker.sock:/tmp/docker.sock:ro"
        - "/etc/nginx/certs"

    letsencrypt-nginx-proxy-companion:
      image: jrcs/letsencrypt-nginx-proxy-companion
      volumes:
        - "/var/run/docker.sock:/var/run/docker.sock:ro"
      volumes_from:
        - "nginx-proxy"

The first service called proxy is the Alfresco Proxy which is used to access Alfresco applications such as Share or Alfresco Content App. The following two services together form the Docker Companion Images for SSL certificate management with Let's Encrypt.

Testing

I had a great desire to try out the new build engine GitHub Actions to test the issuance of the SSL certificate for the domain a.notreal.com I thought it should be possible to run the GitHub Action Runner directly on AWS instead of GitHub itself. I could then use the CNAME record to point to the Public DNS name of the AWS Runner. And yes, that actually worked. A lot of help was given to me this blog post by Lothar Schulz which describes exactly how I can use AWS Ec2 VMs as GitHub Actions Runner. My GitHub workflow looks like this:

name: Test Deployment

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:

  linux-build:
    # runs-on: ubuntu-latest
    runs-on: self-hosted
    strategy:
      fail-fast: true

    steps:
    - name: Checkout
      uses: actions/checkout@v2.1.1
    - name: Setup Node.js environment
      uses: actions/setup-node@v1.4.2
    - name: Execute start.sh
      run: |
        mkdir -p data/solr-data
        mkdir -p logs/postgres
        sudo chown -R 33007 data/solr-data
        sudo chown -R 999 logs/postgres
        chmod +x start.sh && ./start.sh -spr https -sh a.notreal.net -sp 443
    - name: Docker Compose deployment has failed
      if: ${{ failure() }}
      run: |
        docker-compose config
        docker-compose logs
    - name: Execute Postman Tests
      run: |
        npx newman run test/alf-ec2.postman_collection.json -e test/alf-ec2.postman_environment.json -r cli,json --reporter-json-export tmp/newman/report.json
    - name: Postman has failed!
      if: ${{ failure() }}
      run: |
        echo "Postman Failed!"
        cat tmp/newman/report.json
        docker-compose config
        docker-compose logs

In the workflow file of GitHub Actions we see that Alfresco is started with the domain that points to the EC2 Runner via the CNAME record. By executing 'start.sh' Alfresco is started and at the end the availability is checked. Now we should just make sure that the index engine Solr is also working. Using Newman, which is a CLI tool from Postman, a request with subsequent validation is used as a test:

protocol=https
host=a.notreal.net
port=443
POST {{protocol}}://{{host}}:{{port}}/alfresco/api/-default-/public/search/versions/1/search

This tests the accessibility of Solr. At this time I don't see much point in more request tests with Postman. But it would be no problem to do this at a later time. Are you more interested in the possibilities with Postman? Some time ago I wrote an interesting article where I also use Postman.

Summary

Encrypted connections to the Alfresco proxy are essential for a production environment using Alfresco. It requires a lot of manual effort to create, authorize and renew the required SSL certificates. With the great and free offer from Let's Encrypt this effort can be reduced to almost zero. If you find Kubernetes as exciting as I do, you might consider creating small Let's Encrypt charts which can be used in a Kubernetes deployment to achieve SSL certificate management in a cluster.

If you use Let's Encrypt for a production environment, please consider making a donation and Let's Encrypt. This way we can guarantee that this service will remain free in the future. Thank you for your attention and I hope for a lot of feedback :).

Become a Patreon!

The Alfresco Provisioner

Tue, 19 May 2020 00:00:00 GMT

Hi Alfrescans.

Have you ever installed Alfresco on an AWS Ec2 instance? Maybe even with Docker Compose? Even though Docker has made installing Alfresco easier, it is still a lot of work! First af all you have to create an Ec2 instance. To do this, you have to log into AWS, select an Ec2 type and start it. Then connect it there, install Docker and then you have to do an Alfresco Docker Compose installation. How great would it be if we had a Managed Service that did all this for us for us?

Good news! That's exactly what I've been working on in the past few weeks. As a result of my work, all you will have to do in the future is to enter the desired Ec2 type and choose the Alfresco Docker Compose Deployment and the Alfresco Provisioner does the rest and installs Alfresco for you! This allows everyone to quickly and easily create Alfresco backends for themselves and their community. In the next sections, I will explain in more details how it works and which technologies I used.

For Whom?

I see a clear need for the Alfresco Provisioner amongst smaller groups who could profit from the use of Alfresco products I already have a good real life example from my local area. I am a member of a little church community. Now the church community would like to modernize, also towards digitalisation. For example, videos of church services are uploaded to YouTube or pictures taken end up somewhere in Dropbox. Community projects are mainly planned only on pen and paper, because of the right digital tools are simply missing. I see Alfresco products being able to help a group like my church community to manage content more efficiently. You are also likely to know groups such as clubs in your area that could benefit from Alfresco.

The rapid provision of the Alfresco backend for such communities could also help with the mandatory digitalisation during the corona crisis. It should be avoided as much as possible to exchange real documents from person to person or to work on them face to face together. I would like to do my part to help with the pandemic and maybe the Alfresco Provisioner is a possibility.

It will be challenging to keep the costs as low as possible and to pass on the cost advantages of the cloud to the user in the best possible way. A few ideas that I already have in this regard are the use of Reserved Instances, Spot Instances, S3 as object storage, Pay as you go for the Alfresco instances and the possibility to use your own Ec2 instances.

Another possibility would be to create the Docker Compose and use easily expandable backends for demo purposes and thus show potential users what Alfresco products are capable of.

Technologies

The Alfresco Provisioner uses a variety of modern technologies. I describe them below in the categories - backend, frontend, and DevOps Pipeline.

Backend

The Alfresco instances are operated with the help of Docker Compose, which I created with the Alfresco Docker Installer. These then run on AWS Ec2 instances. The use of Ec2 instances for an Alfresco backend has become very common. However, I make this creation of the instances as dynamic and simple as possible. I have created an AWS Api GateWay which is autonomously able to start, stop and terminate Ec2 instances using API requests from the user.

After the instance is started, the user receives a url that brings him to the Alfresco backend. A number of lambdas, step functions, dynamo DB tables and S3 buckets also work behind the Api Gateway to make them possible. All of this is written with infrastructure as code in the AWS CDK. I use TypeScript as the programming language for the CDK and the Lambdas. I began to love TypeScript during this project and it enables me to develop the Alfresco Provisioner very quickly. A Swagger file is also created using CDK, which is used as the UI and client library for the front end. I have more detailed information on how to seamlessly integrate Swagger / OpenApi in CDK described in this article.

Frontend

The frontend is still very modest compared to the backend. This is partly due to the fact that I still have a lot to learn in the frontend area, but also because the backend is developing quickly and I don't want to readjust UI components every time. My choice of technologies are React and TypeScript. React impresses with a rich selection of beautiful components and TypeScript is an ingenious language. To briefly describe TypeScript it only takes the best out of Java and JavaScript. What I like most on TypeScript is the usage of Types. This serves as documentation and I create order in the rather typeless jungle of JavaScript. TypeScript's zero handling or undefined handling is also great.

In addition, I decided to use the OpenAPI UI Explorer as my UI. This simply has the great advantage that the UI components are only rendered based on the Swagger file and I don't have to do it myself. Quasi automatically generated components. You can find out more from my previous posts and on the Swagger UI page in GitHub and on Swagger.io

DevOps Pipeline

By DevOps Pipeline I mean the process from implementing new features to bringing those into production. There is a huge variety for archiving that!. In general, however, it has been set to be automated as much as possible. I have achieved a very high degree of automation in my pipeline. I just have to push to the master branch, then a test stack is set up and postman tests are carried out against the API GateWay. If the tests were successful, the production stack will be updated automatically. I can easily influence this behavior in the code itself by setting the following variables:

env:
  destroyBefore: true
  deploy: true
  updateProduction: true
  destroyAfter: true

I also use AWS CDK which allows me to define the test and production stack using TypeScript. More about this can be found in my previous post CDK Multistack or CDK in general. I find Travis and AWS CDK a brilliant combination

How does it work?

The provisioning of the Alfresco instances takes place via the REST API. The user logs in to the website of the managed service, then receives an OAUTH token and can then use it to create new Alfresco instances, call up and change old ones. By using the OpanAPI UI Explorer, the user can explore the very detailed information on how to use the REST API, i.e. which parameters in the requests and response are to be set or expected.

Below I list how the body in the POST request is for creating a new instance:

{
 "userId": "alfresco",
 "alfType": {
  "ec2InstanceType": "t2.large",
  "gitRepo": "alf-ec2-1",
 },
 "customName": "Alf Backend 1"
}

The userId is just the username in the system and customName a nickname for the Alfresco backend created with the provisioner. AlfType describes the type of Alfresco backend. This consists of the type Ec2 VM ec2InstanceType and an Alfresco Docker Compose Repository gitRepo. The GitHub repository is on my account and private. I currently only support one github repo:

alf-ec-1: ACS 6.2 Community, ACA

All components were installed with the Alfresco Docker Installer. Of course, this offer is planned to be expanded widely in the future. It would also be conceivable to use Alfresco Git repositories from other users who are not from my GitHub realm. You just have to meet a certain standard and then there is nothing to be said against it.

A detailed description of all REST endpoints as well as their requests and response parameters can be found on the Alfresco Provisioner UI page.

Planned Features

Already many features are planned, which enable the user to save money, facilitate the handling of Alfresco products and make the backend more secure. The user should be able to stop and start the instances. If the instance is stopped, the user no longer has to pay for the computer, but only for the storage or possibly no longer. An extension for the automatic creation of backups would also be useful. For this I will look at Tony's interesting work with Alfresco BART. This would also make it easier to upgrade to other alfTypes. At the moment I have made good progress by writing a tool in Go that can create backups. Now the only thing missing is the possibility to import them back into Alfresco.

No HTTPS has yet been set up for the proxy of the instances. I plan to use the certificates provided by Let's Encrypt. For this you simply have to create a Docker Companion, which takes over the certificate management. It should also be possible to have the certificates renewed automatically before expiry. So far only ACS Community is set up, but I plan to offer other Alfresco products such as ACS Enterprise or APS. I'm also thinking about an onboarding layer for ACA.

Another useful feature would be if the user can simply redirect the url instance to his own domain url.

Closed Alpha

I will soon make the Alfresco Provisioner available to the public. For that I thought to organize a closed alpha first. Interested people can try out the Alfresco Instance Provisioner and provide feedback and more.

If you are interested, write to me and I will then create your access accounts. All I need is the email address and the desired user name. I don't have a specific launch date for the Closed Alpha yet, but I am confident that I will do it in May.

Brief instructions on how to use the Alfresco Provisioner

I decided to make a short video about how to use the Alfresco Provisioner. The video can be viewed here: https://www.youtube.com/watch?v=OOEOE_ncsx4

More about the Alfresco Products

To give you a quicker

From a Personal Perspective

What I like most about this project is the fact that most of the implementation, testing, development and documentation is done or supported using the Swagger definition. This goes from the validation of the request parameters, implementation of the REST interface, documentation, visualization with Swagger UI and automated tests in Postman. I love the approach with the Swagger definition, which I can preview using the Swagger plugin in Visual Studio Code, to use for almost everything. For sure, that's the future of server implementations.

Furthermore, I'm just still overwhelmed with what a single developer is able to do with AWS CDK, especially in the TypeScript flavor. I have already described my enthusiasm for CDK here and do not want to repeat it here. Please catch up.

Summary

What started as a simple AWS CDK experiment quickly developed into a great new service that I called the Alfresco Provisioner. This makes it easy to provide Alfresco backends that were developed with Docker Compose. In the closed alpha phase I cordially invite you to use the service.

Kudos

A big thank you to the AWS CDK Community in Gitter. I have rarely seen a community so in love with technology.

Become a Patreon!

Alfresco Virtual Hack-a-thon May 2020 Summary

Fri, 15 May 2020 00:00:00 GMT

Hi Alfhackas.

The Alfresco hack-a-thon was great. What I liked most were the personal conversations with many great ex-colleagues and other Alfresco folks. In addition, I was able to learn valuable information about planned projects within Alfresco. For example, I am very excited to see where Digital Workspace, the beautiful webapp for ACS Enterprise, will develop. Very soon it will get UI elements to display APS elements like tasks and processes. I was also advised that as an Alfresco partner I could get access to the Digital Workspace Repository, which I gratefully will do.

Lets Encrypt

My project for the Hack-a-thon was to give the Docker Alfresco Installer a Docker extension, which would enable Lets Encrypt to use authorized certificates for the HTTPS connection. Even the extension would manage the entire life cycle of the certificate. I haven't completely finished with it, but I was able to implement a good bit and I'm guaranteed to finish it this weekend.

Conclusion

Many thanks to the organizers of the Alfresco Hack-a-thon and to the great Alfresco community. I will definitely be back at the next Hack-a-thon. Cheers.

Become a Patreon!

Alfresco Virtual Hack-a-thon May 2020

Fri, 08 May 2020 00:00:00 GMT

Hi Alfrescans.

On May 11, 2020, Alfresco will host a virtual hackathon. You can find the current project ideas here at hub.alfresco.com. With this post I want to invite you to participate.

Why Participate?

There can be several reasons why you should participate. If you like the Alfresco products, it's great to help develop the products further or even create completely new ones. For example, I would like an SSL Companion Container for the Alfresco Docker Installer. Without further ado, I simply registered the project to help organize and implement it.

It is also a great opportunity to make contacts with motivated developers from all over the world. During my time at Alfresco, I got to know smart and helpful developers, from whom I learned a lot. I am always looking for opportunities to learn and the hackathon is the perfect opportunity for that.

My company would also like to organize a hackathon in the future and the Alfresco Hackathon could be a perfect opportunity to learn how to organize such hackathons.

The best reason: why not? It could be an interesting new experience for you and if you don't like it you can go at any time and just don't take part next time.

Summary

The Alfresco Hackathon on May 11, 2020 is just around the corner. In the starting blocks my friends :)! After the event I will report in the form of a post here how I liked the hackathon. By 11 then and stays healthy :D!

Become a Patreon!

AWS CDK Multistacks for Test and Produktion

Sun, 19 Apr 2020 00:00:00 GMT

Hi AWS Fans,

Last time I described how well AWS CDK is suited for describing infrastructure as code. This article is about how CDK could look in a test and production environment. In my private project I have already implemented something like a production environment and I am very enthusiastic about it. In short, I use CDK's Multistack. In the next sections I describe how my deployment looks like.

CDK Multistack

A CDK multistack is a CDK deployment that manages multiple stacks. These stacks can be in the same or different regions, or even in a different AWS account. The different stacks can then be easily initialized with different parameters before creation. Here is an example with two stacks:

new MultiStack(app, "EuWest2Prod", {
    environment: 'prod',
    env: {
      region: "eu-west-2",
      account: 'ABC'
    },
    // disable create ec2 instance
    // createInstances: {
    //   imageId: 'ami-04d5cc9b88f9d1d39'
    // },
    cognito: 'true'
    domain: {
      domainName: 'api.nope.dev',
      zoneName: 'api.nope.dev.',
      hostedZoneId: 'AA',
      certificateArn: 'arn:aws:acm:eu-west-2:ABC:certificate/xyz'
    }
  });

new MultiStack(app, "EuWest2", {
  environment: 'dev',
  env: {
    region: 'eu-west-2',
    account: 'XYZ'
  },
  createInstances: {
    imageId: 'ami-0cb790308f7591fa6'
  }
});

This is a small excerpt from my project. The first stack called EuWest2Prod describes the production stack, which is even in another account. The second stack EuWest2 is the test stack. This type of parameterization can be used to turn features on and off. In the production stack, by commenting out createInstances, I disabled the feature to create Ec2 instances in order to save costs. The parameterization also supports simple migration from one AWS account to another.

In the next sections, I want to talk more about my test stack and production stack. But I have to mention fairly that I don't have a real production yet. So to speak real customers who use my service. I hope to be able to change this soon in a closed beta version. At the moment I am trying to gain experience on how I could keep a real production running as good as possible, i.e. implement seamlessly new features. AWS ChangeSets for cloud formation stacks are very suitable for that. These only change the respective stack with the necessary updates. Before you continue reading, I definitely recommend to learn about AWS ChangeSets for CloudFormations.

Test Stack

Test stacks should be primarily cheap. The tests are carried out and the log result should be easily accessible to the developer. Shortly afterwards, the stack should destroy the resources to save money. If a manual assessment of the test stack is still desired afterwards, a delay in destruction can be implemented. I did that for the Ec2 instances which terminate themself after 55 minutes.

And of course the test stack should test. What exactly that is depends on your use case. I test against an API GateWay and its endpoints. In the background, several DynamoDB tables and EC2 instances are created or terminated using step functions. I do that with Postman. If you are more interested in testing, I have already written about it in one of my previous post.

In itself, that's it. If the test phase was successful, the update can be applied to production. This is described in the next section.

Production Stack

The production stack is intended for customers. Everything has to work as well as possible there and new features should only be implemented after they have been proven in the test stack. It can also make sense to operate the production stack with slightly changed service settings. An example of this would be that I don't really need user administration with Cognito in the test stack, but I do in the production stack. This keeps the complexity in the test stack lower. In the snippet above, I defined cognito in production. Another example would be EC2 instances. In the test stack, low cost EC2 types are fine, but stronger and therefore more expensive instances are required in production, usually.

The question now is where the production stack should be deployed. Fortunately, CDK gives us every freedom there. It is possible to deploy the stack in the same or in a different region, in the same account. The production stack can also be deployed in another account. Profiles have to be used for this. Here is an example:

cdk deploy "$ STACK_NAME_PRODUCTION" --profile = prod

This stack is now deployed in another AWS account. So far I have had a good experience with this type of deployments. However, this type of deployment has one disadvantage. Since I am now dependent on the Profile parameter, I cannot use the same cdk deploy command for the test stack and I have to execute two CDK commands:

cdk deploy "$ STACK_NAME_TEST"
cdk deploy "$ STACK_NAME_PRODUCTION" --profile = prod

It would be great if I could simply specify the profile name when defining it in the multistack. This disadvantage is of course not really bad, since in a proper pipeline the test stack should run first and after passed test an a manual approval the production stack is deployed.

EDIT: I received feedback that there is already a workaround for this in the npm registry: https://github.com/hupe1980/cdk-multi-profile-plugin

DevOps Travis Pipeline

Every developer should be familiar with Travis. The Travis Free Tier allows you to use VMs from Travis if your GitHub project is public. These then only have to be defined and configured in the .travis file. That's exactly what I'm doing. To be honest, I use Travis a lot. I just briefly describe my workflow when I create a new feature:

Write code adjustments of new feature in master.
If necessary add new test in Postman.
master commit -> Travis build is triggered.
Travis fails or passes.
If it failed, the stack reverted automatically and I evaluated the logs to see what went wrong. Back to step 1).
If it passed, the new feature is on all stacks and I can now test it manually.

This is very awesome because I have to do fewer interactions to implement this new feature. The entire build takes about 10 minutes and then I can just do something else for 10 minutes. This type of pipeline has brought me very quick iterations of new features. I am literally flabbergasted what is possible today with just a DevOps. What I can do alone would have required about 10 people 5 years ago instead of just one.

Summary

CDK's multistacks are a great way to manage multiple stacks like a test stack and production stack. For my small project I have great experiences with this type of stack management and described it here. So far, a relatively small Travis build has been sufficient to create a useful pipeline. I am considering looking into AWS CodePipeline to see if it could benefit me. I hope you enjoyed my little summary :)

Become a Patreon!

ACS and APS Deployment with Docker Compose

Sun, 12 Apr 2020 00:00:00 GMT

Hi Alfrescans.

It is time again to report about an exciting Alfresco partner project from me. For a customer here in Germany I am developing a POC which should use ACS 6.2 and APS 1.10. After a bit of persuasion, I was able to convince those involved to use Docker for ACS 6.2. My plan was to create a Docker Compose Deployment which included ACS, APS and openLDAP as an identity provider. The complete deployment can be viewed in GitHub. One thing should be mentioned in advance. Alfresco officially recommends to use Kubernetes for such deployments, but the cloud is a taboo topic for my customers so far. I am also excited to see how far I will get with Docker Compose for such a deployment.

Docker Compose Setup

To start, I will describe in more detail the technologies used in my deployment. As indicated at the beginning, ACS 6.2 should be installed. The most recently released version is 6.2.0.3. So that has to be reflected in the alfresco Dockerfile at /alfresco/Dockerfile. Share remains on version 6.2.0 since no newer version has been released. ACS is to be connected with an openLDAP DB for user provisioning. Fortunately, there are already great images for creating and managing an openLDAP DB. For the visual administration of the openLDAP DB I use phpldapadmin.

APS 1.10 should be used to model the workflows. The fact that Alfresco no longer offers Docker Compose Reference Deployments for APS 1.10 is somewhat unfortunate. I guess that is because of the Kubernetes deployments and that there is a stronger focus on those. Fortunately, there are still a few old Docker Compose templates and with a few modifications they work too!

Since it is very memory intensive to run ACS and APS at the same time, I decided to split the complex deployment into three separable deployments. So I can continue to work efficiently on my 16 GB memory laptop. The division is done using Docker Compose files. The first is the ACS deployment, followed by the second with the APS deployment. Finally, there is ACS and APS deployment. I think that's a brilliant idea because if I just want to work on ACS or APS alone, I don't have to start up the entire stack. In addition, I am considering deploying the third deployment ACS and APS in EC2 using a good looking DevOps pipeline.

Prerequisites

It is clear that you need Docker. If you are working on a Windows or Mac, I recommend the Docker Desktop. After the installation please remember to make more memory available to the Docker environment as the default 2 GB are far too little to get ACS up and running. At least 12 GB would is recommended!

The ACS Docker Compose File also uses private images hosted by Alfresco in quay.io. For these, it would be important to request access data from Alfresco Support. If you have this, just run the following command and enter the credentials:

docker login quay.io

More or less optional but my example in the GitHub repository uses Node's NPM for a CLI script tool to wait until Alfresco has finished booting. So it's best to install the node and NPM as well.

Git Prepare

Of course, a git repo has to be created first. I called mine alf-acs-aps. As in my last Alfresco Docker Project I recommend using the Docker Alfresco Installer to prepare the empty Git repository. I describe exactly how the Docker Alfresco Installer can be used here. When installing, make sure that LDAP and an SMTP server are also selected. If you are not using my GithUb Repo, you should now integrate the APS 1.10 services into the Docker Compose Deployment. I recommend that you split the deployment into three parts. All three are explained in the next sections.

ACS Deployment

First a docker-compose-base.yml file is created which contains all the services required for all three deployments. In my case this is an openLDAP and postgres database. As well as a mail server. Then the ACS Docker Compose File is created. The entire deployment can now be started as:

docker-compose -f docker-compose-base.yml -f docker-compose-ACS.yml up -d --build

and so stop:

docker-compose -f docker-compose-base.yml -f docker-compose-ACS.yml stop

If a complete ACS restart with empty databases and other storage is desired, simply execute these commands:

docker-compose -f docker-compose-base.yml -f docker-compose-ACS.yml down
rm -rf data
rm -rf logs

APS Deployment

The APS deployment is started as:

docker-compose -f docker-compose-base.yml -f docker-compose-APS.yml up -d --build

and it is stopped like this:

docker-compose -f docker-compose-base.yml -f docker-compose-APS.yml stop

To erase just do the following:

docker-compose -f docker-compose-base.yml -f docker-compose-APS.yml down
rm -rf data
rm -rf logs

ACS and APS Deployment

Warning a warning in advance! This deployment is very memory intensive. Your laptop or PC should have at least 16Gb. Alternatively, I plan to deploy this variant in the cloud on EC2.

And this is how all services are started:

docker-compose -f docker-compose-base.yml -f docker-compose-ACS.yml -f docker-compose-APS.yml -f docker-compose-Proxy.yml up -d --build

and so stopped

docker-compose -f docker-compose-base.yml -f docker-compose-ACS.yml -f docker-compose-APS.yml -f docker-compose-Proxy.yml stop

And just type to tear it down:

docker-compose -f docker-compose-base.yml -f docker-compose-ACS.yml -f docker-compose-APS.yml -f docker-compose-Proxy.yml down
rm -rf data
rm -rf logs

Outlook

As indicated, I would like to bring the third deployment ACS and APS to AWS EC2. Preferably even in a nice DevOps pipeline, i.e. commit after master triggered and deploy to EC2, where automated tests are then carried out, such as whether ACS and APS have successfully booted.

SSO (Single Sign On) allows you to only have to register once for Alfresco products. That means if I log in to Share for example and then open Alfresco Digital Workspace or the Ativiti app, I don't have to log in again. To do this, AIMS (Alfresco Identify Manager Service) would have to be configured with ACS and, in addition, the SAML amp.

Summary

You don't always have to shoot with the Kubernetes shotgun when it comes to container orchestration. Docker Compose can also be used to orchestrate containers in a simplified manner. There is even the option of dividing complex deployments like that into smaller pieces. This not only saves memory footprint, but is also excellent for fast iterations during development. So feel free to try it out yourself and deploy ACS and APS using Docker Compose. If there are any questions, just write to me.

Kudos

To Thijs from the amazing Alfrescan Discord Channel for suggesting this kind of Docker Compose deployment split.

Become a Patreon!

AWS CDK Api Gateway with Swagger

Sun, 05 Apr 2020 00:00:00 GMT

Ahoi AWS'ler and Swagger fans

In the last post I showed how AWS CDK can be used as a welcome alternative to YAML for describing your cloud infrastructure. Working on my CDK project from I encountered a problem handling Swagger files. But first I want to explain what the Swagger definitions are. It should also be said that the new version of Swagger has been called OpenApi since 2018. When using AWS API Gateway, it is convenient to use Swagger for parameter validation such as Query, Path and Body Parameter. What exactly Swagger is and why I think it's so great I describe in the next section.

What is Swagger

Swagger is a YAML or JSON template language for describing RESTful APIs. Following I describe what is great about Swagger. First of all, the templates are extremely suitable as documentation about the API itself, because a stylish looking HTML UI can be generated out of the template, which describes the API endpoints very well. Such an UI can be seen in the cover of this blog post. Even more brilliant is the UI that can be used directly to test the endpoints, i.e. to send and receive requests and responses. Many API interfaces, like the AWS API Gateway, offer the parameter validation of requests with using Swagger. I give a little example for what I mean by parameter validation using the following example:

parameters:
    - in: query
    name: userId
    description: Get items of that user
    required: true
    type: string

Here you can see a parameter of the type query. That means it would look something like this in the URL

http://<url>/items?userId=martin

With parameter validation, I can then define certain properties of the parameter, such as the name userId, whether it is required and what type the value should be, in our case as of the type string.

Another very powerful feature is that it is possible to generate client libaries from the Swagger definition (https://swagger.io/tools/swagger-codegen/). For example, Alfresco does this with the API Explorer (For more information, see GitHub Api-Explorer) and ADF (or ADF JS Github). There, for example, the Swagger File becomes the JavaScript API Library generated, which can be used as a wrapper for the API requests and is also in ADF Components Github.

It's also cool, Postman offers an import feature for Swagger definitions. Then a collection is created immediately. This is very handy if you want to start writing the requests in Postman.

AWS Api Gatway with Swagger

All the features from the previous section sound very tempting, don't they? How nice that AWS API Gateway also offers to extract Swagger definitions from its deployments and also to import it using CloudFormation or CDK. Unfortunately in connection with AWS CDK, extracted Swagger files are not so easy to reuse for their description. The biggest problem is that if, for example, a lambda is updated, its ARN also changes. However, the Swagger Files extracted from the API Gateway use exactly these ARNs when Lambda is used as a backend implementation for an endpoint. And since the ARN changes, they are outdated in the swagger file. Below is a Swagger Snippet and there you can see the Arn I'm talking about behind the uri keyword.

paths:
  / items:
    get:
      responses:
        '200':
          description: Ok
          content: {}
        '401':
          description: Authorization information is missing or invalid
          content: {}
      x-amazon-apigateway-integration:
        uri:> -
          arn: aws: apigateway: [secure]: lambda: path/2015-03-31/functions/arn: aws: lambda: [secure]: [secure]: function: ApiLambdaCrudDynamoDBExam-getAllItemsFunction0B7A9-1KKCGIJ01C3NI/invocations
        passthroughBehavior: when_no_match
        httpMethod: POST
        type: aws_proxy
      parameters:
        - $ ref: '#/parameters/alfUserIdParam'
      x-amazon-apigateway-request-validator: Validator

The AWS CDK team knows about this problem and it is on the roadmap. So hopefully if you read this post the following workaround is no longer necessary. For completeness, I list the related issues here:

https://github.com/aws/aws-cdk/issues/723 https://github.com/aws/aws-cdk/issues/1461

Workaround

To solved the problem with deploying CDK twice. The first deploy does not use a swagger definition but the second does. It is really interesting how I generated the Swagger file. In the following I try to describe the process, but please also look into my GitHub repo (there in .travis) for how I implemented this workaround.

I deploy or Swagger File with:

export WITH_SWAGGER = 'false' && cdk deploy --require-approval never

WITH_SWAGGER is an env variable in the CDK Deployment File index.ts. This means that no swagger file is used programmatically for the deployment.

Now I can use the AWS CLI to extract the swagger file from the API gateway:

aws apigateway get-export --parameters extensions = 'integrations' --rest-api-id $ REST_API_ID --stage-name prod --export-type swagger --accepts application/yaml tmp/swagger_new.yaml

The generated Swagger File tmp/swagger_new.yaml looks something like this:

swagger: '2.0'
info:
    version: '2020-03-29T16: 59: 22Z'
    title: Items Service
host: hjtiuj2ou1.execute-api. [secure] .amazonaws.com
basePath:/prod
schemes:
    - https
paths:
    / items:
        get:
            responses:
                '200':
                description: Ok
                '401':
                description: Authorization information is missing or invalid
            x-amazon-apigateway-integration:
                uri:> -
                arn: aws: apigateway: [secure]: lambda: path/2015-03-31/functions/arn: aws: lambda: [secure]: [secure]: function: ApiLambdaCrudDynamoDBExam-getAllItemsFunction0B7A9-1MLKSKO1RUL3I/invocations
                passthroughBehavior: when_no_match
                httpMethod: POST
                type: aws_proxy
            x-amazon-apigateway-request-validator: Validator
        post Office:
    ...

It looks very good. However, the parameter validation is still missing. For this a second Swagger definition has to be created manually templates/swagger_validations.yaml and looks something like this:

paths:
    / items:
        get:
        parameters:
            - Name: alfUserId
            in: query
            required: false
            type: string
            $ ref: '#/parameters/alfUserIdParam'

What we want now is a merge of both swagger files. NPM offers a useful tool for merging YAML files:

npm i -g merge-yaml-cli

The Swagger definition tmp/swagger_full.yaml now contains the current LambdaUri and the parameter validation:

swagger: '2.0'
info:
  version: '2020-03-29T16: 59: 22Z'
  title: Items Service
host: hjtiuj2ou1.execute-api. [secure] .amazonaws.com
basePath:/prod
schemes:
  - https
paths:
  / items:
    get:
        parameters:
            - Name: alfUserId
            in: query
            required: false
            type: string
            $ ref: '#/parameters/alfUserIdParam'
        responses:
            '200':
            description: Ok
            '401':
            description: Authorization information is missing or invalid
        x-amazon-apigateway-integration:
            uri:> -
            arn: aws: apigateway: [secure]: lambda: path/2015-03-31/functions/arn: aws: lambda: [secure]: [secure]: function: ApiLambdaCrudDynamoDBExam-getAllItemsFunction0B7A9-1MLKSKO1RUL3I/invocations
            passthroughBehavior: when_no_match
            httpMethod: POST
            type: aws_proxy
        x-amazon-apigateway-request-validator: Validator
    post Office:
    ...

Now this swagger file can be used in the second CDK deployment:

export WITH_SWAGGER = 'true' && cdk deploy --require-approval never

Attention!: Always take care with creating a new stage deployment in AWS API Gateway. I do this in Travis with the AWS CLI:

aws apigateway create-deployment --rest-api-id $ REST_API_ID --stage-name prod

Summary

Swagger Definitions are ideal for creating, editing, documenting and testing APIs such as those from AWS Api Gateway. They are not only useful directly in the backend. As well they are suited in the frontend because Swagger can generate amazing looking and useful Swagger UI to understand the backend better in a way shorter time.

Become a Patreon!

AWS CDK Tutorial with Travis Deployment

Sun, 29 Mar 2020 00:00:00 GMT

Ahoy AWS folks

For a private project I have been working on an AWS CDK example in the past few days. AWS offers a range of great AWS CDK examples. The experience I had was so good that I decided to write a blog post about it. I was fortunate to work with CloudFormation very early in my AWS career. It was a lot of fun and amazing to experience infrastructure as code at first hand. Now there seems to be a new kid on the block called AWS CDK (Cloud Development Kit), which can preferably being used as a language to describe the infrastructure in AWS. CDK promises to be a welcome alternative to CloudFormation, instead of writing rather complex YAML, the CDK supports programming languages such as Java, Python, JS or Typescript to define the infrastructure. I think that's a great idea because if you are using like already Typescript to write your Lambdas, you can use it as well for CDK. My example code CDK Example Repo is on GitHub. So feel free to use it. In the next sections I will briefly describe the steps I took to create that Repo.

The Requirements

If you want to try AWS CDK, you need an AWS account. You also need AWS CLI credentials that allows your deployment to use resources in your AWS account. Usually you create an IAM User. It is important that you save the access data, i.e. the access key ID and the secret access key. These will later be required in the deployment. The AWS example uses NPM as the package manager. That means it takes care of the dependencies such as the aws-cdk which is installed with the following command:

npm install -g aws-cdk

NPM is available here.

Prepare the Git Repo

So now lets start. First we create our GitHub repository which is still completely empty. For me I called it cdk-example. Then the AWS CDK Examples repo should be forked and checked out locally. With the intention that we can do cherry picking more easily this way and simply copy over example code to our repository.

If you look at the AWS CDK Examples Repository you might are a bit overwhelmed of the choices. I myself can recommend the examples from the Typescript category because they are many and Typescript seems to be becoming increasingly popular. I also recommend starting with the Api Cors Lambda Crud DynamoDB Example because it already contains a lot of AWS services like API Gateway, Lambda and DynamoDB. That example is a simple CRUD storage which is backed up with DynamoDB. +

It's best not to change the existing source code and first try to get it working. I myself had a little difficulty. The example has a missing dependency. The NPM uuid library, which is used to generate IDs, is missing. To do this, change to the src/ directory and execute the following commands:

npm init
npm install uuid

The code also needs to be adjusted a bit, as the uuid has changed somewhat with the versions. Alternatively, you can simply copy from my published GitHub Repo. In the next section we will get the example up and running.

Manually Deploying CDK

In the previous section we have prepared the repo so far, now of course we want to deploy it! Since we have our deployment in AWS, it must be ensured that the access credentials are set correctly. In the This is required we have already received the credentials. There are different ways to make these credentials available to your build. The automated deployment in Travis, explained in the next section, works with environment variables. For manual deployment it is more convenient to create the credentials in the aws config folder. simply insert the following files:

~/.aws/conf and ~/.aws/credentials too! For Windows users simply use your user folder instead of the ~. Write done the following content:

[default]
aws_access_key_id = AKI ...
aws_secret_access_key = for you ...
region = eu-west-2

You should choose a region that is close to you and offers a lot of functionality. I am from Germany and the Frankfurt Region is more close, that region is often behind with new features, I chose the London region eu-west-2.

Now switch to the repo and execute the following commands:

npm install -g aws-cdk
npm install
npm run build

The npm install commands are loading the dependencies and just needed to be executed rarely. The run build command, however, whenever changes have been made. An interesting fact is that it converts the TypeScript files into JavaScript files. Since TypeScript is a superset of JavaScript, you do not have to reinvent TypeScript tooling and simply use the existing JavaScript tooling out there. Anyway take care and not get confused about those generated JS files. You might apply changes there and wondering why it is not reflected in the build, as I did.

Next, CDK must be configured so that it knows which account and which region to use. I don't understand why CDK cannot generate this from the AWS folder. Then CDK can be deployed:

cdk bootstrab aws//{account}/{region}
cdk deploy

Also interesting with:

cdk synth > cfn.yaml

Can you look at the generated CloudFormation template.

Automation with Travis

The manual effort for deployment certainly is still low, but this will change as complexity increases. So I had great experiences to automate as much of it as possible. And now imagine how cool it is to change the production environment with just a push to master. I describe a little foretaste of what this could look like exactly here. Also, I would like to say again that you could look at my Travis definition here CDK Example Repo.

What I do there in Travis is explained quite simply. I just do the manual deploy steps from the previous section in Travis. Since Travis has my AWS credentials in the form of encrypted environmental variables, I allow it to be manipulated in my AWS account resources. The following environmental variables were required: AWS_ACCESS_KEY_ID, AWS_ACCOUNT_NUMBER, AWS_DEFAULT_REGION, AWS_SECRET_ACCESS_KEY, CDK_DEFAULT_ACCOUNT, CDK_DEFAULT_REGION.

Just not having to do the manual deployment anymore may not seem very useful at first glance, but believe me, it's worth it! After the CDK deployment is finished, Postman tests are then carried out in the form of requests and response validations. I will describe more about this in the next section.

Stack Testing with Postman

I think everyone knows about Postman now the days. It is a glorious tool for testing APIs that work with requests and responses. In my case it is a REST API using AWS API Gateway. In my GitHub repository I have put together a small collection of requests and response tests, which Postman calls collections. These can then simply be used with the CLI Newman to execute the collections created with Postman. Of course, Newman must then be installed on the build machine beforehand. In my repo I do this via NPM in package.json. Simply run the following command:

npm install newman

The creative process of writing collections is firstly you cerate a new collection in Postman and than you write request. The request runs against the REST API and returns a response. The response can then be validated. Postman offers a wide range of options for validating responses. So you can easily compare the status code in the response with an expected value, compare other expected values, create variables for next requests and much more. A tip from me, try to keep the amount of tests low.

Summary

AWS quite new CDK is a great tool for creating AWS infrastructure. It is amazing that you can write in the same language that you would use for your Lambdas. Combined with automatic deployment and testing like with Travis, changes to a possible productions stack are easy and make fun. I also believe that this means that there is no need or less need to test Lambdas locally. If you need help with your AWS CDK project, don't hesitate to ask me or the helpful CDK Gitter community.

Become a Patreon!

Bash Script Contribution for the Alfresco Docker Installer

Sun, 22 Mar 2020 00:00:00 GMT

Hi Alfrescans,

I made great experiences in my last project with the Alfresco Docker Installer. You can read more about it in my previous blog post. Now I am of course excited to add valuable new features to it. One of them is the Start Script which can be used as a wrapper for the Docker Compose File. In my previous Alfresco Docker Compose projects I always implemented this script because it offers some nice additional functionality. The next section explains what these are.

Features

The most important feature is probably the built-in wait routine. This waits until ACS has finished booting. This is very useful, for example, if you want to run tests after starting ACS with tools like Postman. Data can also be pushed to ACS with the REST API when ACS finished booting. The wait routine is implemented with NPM's wait-on. It makes sense to use the wide range of tools in the public NPM repository, because I then do not have to implement them myself and they are usually well tested.

The next feature is particularly important to me since I am working with Windows. If you try to run Docker Compose with Windows you may get a problem with defined volumes, since they do not support the Windows Path spelling ootb. Fortunately, the Docker Compose Community has provided a solution. Simply set the following environment variable before starting Docker Compose:

export COMPOSE_CONVERT_WINDOWS_PATHS = 1

This is now done beforehand if you start the start script with the -wp flag.

The next features have not yet made it into the Alfresco Docker Installer, but this could change soon. I personally use them a lot in my deployments. It is possible to set the host IP and host port dynamically with the start script using -hi and -hp. I find this very practical when you are in environments that change their IP more often like for example your laptop. At the moment, however, the installer solves this by asking for the host IP and host port during installation. So it is already configurable but only during the installation.

More features are possible but are not currently implemented. E.g. Loading test data with the REST API. In my ADF AI prototype I also have a -b for build the Angular Webapp and an -aca so that only the ACA container is redeployed for fast development with the Ensure ACA web app. In the next section, I will show you how exactly I contributed the Start Script.

Contribution

First, a fork should be created by the Alfresco Docker Installer. Than you make your changes to your fork. In my case it is the start script with the file name start.sh. I saved the script in template/scripts directory and configured the Yeoman setup that the script can optionally be added, very similar to how the LDAP service is added. All changes should be saved to your fork in the form of a remote branch. To do this, a local branch must first be created and changes pushed:

git branch startscript
git add --all
git commit -m "added startscript"
git push -u origin HEAD: startscript

Once all changes have been made and extensively tested, a pull request to the Alfresco Docker Installer Repository can be created. To do this, simply go to the Installer Github page. A Create Pull Request button will appear there.

Oh yeah one thing more. If you planning to contribute a new feature it would be very nice and best practice to add a test which tests the new feature as well. I did that in the .travis file which now tests the deployment and start of Alfresco with and without the new start script. With that it ensures it can work both ways.

Outlook

Next I would like to provide Alfresco's new Alfresco Identity Management Service, also called AIMS, as a contribution. For this, a Keycloak container is added and the Alfresco properties are adjusted. AIMS only makes sense if you also have an identity provider. So AIMS should only be added if LDAP is selected during installation.

Summary

This article describes a brief overview of my first contribution to the Alfresco Docker Installer. I already use the Installer for my projects a lot and contributing to it gave me the perfect opportunity to understand the Yeoman template installer better. The Start Script offers many new additional functionalities. How to do such a contribution has also been described. Do you also have a great idea how to improve the installer? I'm looking forward to hear about it.

Become a Patreon!

Alfresco's Amps and Jars Testing with Docker

Tue, 17 Mar 2020 00:00:00 GMT

Hi Alfrescans,

In this article I will describe how to use Docker to test Amps and Jars extensions for ACS. Share Customizations and Custom Content Models were developed for a customer of OBJECT, which I wanted further develop and test quickly and easily.

Another interesting detail for my project was that the Amps were written for an ACS 5.2.6 deployment and Docker is only supported as a deployment starting with ACS 6.0. So it was relatively challenging to adjust Docker Deployment so that everything worked. If your extensions were written for ACS 6.0 or later, it should be a lot easier.

Why Docker Deployment?

The old fashioned way is to test the locally compiled Amps and Jars on a local ACS deployment, which is configured completely without or almost Docker. That means Tomcat must be installed and ACS with all its dependencies must also be available. Clearly the old-fashioned installer helps, but it is still a time-consuming and unsatisfactory task to install Alfresco like this. In addition, there is the fact that the Amps must always be installed by hand and possibly uninstalled. Not nice!

Docker containers address precisely these problems. With the help of the Dockerfile I can write down which configuration steps should be taken. With Git, deployments can be versioned and reverted. The installation of the Tomcat as well as the configuration of ACS and much more can be written down in these Dockerfiles.

Sounds like a lot of work? Nope! Alfresco maintains ACS Docker Deployments for ACS 6.0 or newer. Such a simple Docker deployment, whereby the containers are orchestrated with Docker Compose can be stowed here for ACS Enterprise and here for ACS Community. The Docker Image for this deployment you can find here ACS Enterprise Image and here for ACS Community Image. These deployments and images are quite limited in point of customizations as we would need for our Amps or Jars. While this is good for a quick start, it does not allow us to test things like our locally created Amps. I describe how this is made possible in the next chapter.

Prepare Git Repo

If the build code of your Amps and Jars are in different git repos, it makes sense to combine them into one to make Docker image creation easier. E.g. the repo and share Amp could be in the same subdirectories. If Maven was used as a build and packaging tool, this is easy to implement with parent poms. After the possible restructuring, the build must be tested if the Amps or Jars are also created. For me that was done with the following simple maven command:

mvn clean install

Now the Docker backend needs to be installed. The orchestration of the containers should be done with Docker Compose. Fortunately, there is the Alfresco Docker Installer which is very easy to use Yeoman supported template installer and already has a wide range of install options such as the ACS version, whether LDAP to be used, etc. offers. I would suggest to first create a folder with the name docker in the Git Repo and to install the Docker Compose backend with the Alfresco Docker Installer as instructed. After installation, the deployment should start with

docker-compose up --build

and the urls can be tested which are:

http://localhost:80  Alfresco Content App
http://localhost:80/alfresco 
http://localhost:80/share

If everything worked, just shut down the deployment with:

docker-compose down

The next section explains how the share or repo Amp / Jar can be integrated into the image.

Customize Share and Repo Images

Now it is time to combine the share and repo modifications into the Docker image using your Amps or Jars. Ideally, the freshly compiled Amps / Jars from the target folder should be used for this. The easiest way to do this is to move the Dockerfile from docker/alfresco to repo/ and docker/share to share/. This allows Docker to access the target directory there when building the image. Now simply set the instruction in the repo and share Dockerfile that the Amps / Jars are copied from the target folder into the image, such as here:

COPY modules/Amps $TOMCAT_DIR/Amps
COPY modules/Jars $TOMCAT_DIR/webapps/alfresco/WEB-INF/lib

COPY target/myAmp-62-repo-1.0-SNAPSHOT.Amp $TOMCAT_DIR/Amps/myAmp-62-repo-1.0-SNAPSHOT.Amp

Now the docker/docker-compose.yaml file has to be changed and the directories for the repo and share image have to be adjusted. In my setup it looks like this for repo:

context: ./../repo

And that for Share:

context: ./../share

Amps / Jars for ACS 5.2

If you want to test Amps or Jars for an ACS 5.2 deployment like me, there are a few difficulties. It may happen that the dependencies used for the 5.2 Amp / Jar are no longer compatible with the ACS Docker Deployment. It will be noticed that errors are thrown at the booting time of ACS. The first step is to find out which dependencies (usually Jar files on the Tomcat classpath) are causing the problem. It is then recommended to create a Maven profile dedicated for the 6.2 Amp, which then reloads the dependencies with the correct version.

Then the conflicted dependency must be also deleted in the Dockerfile. Don't be discouraged, it can be hair-raising and time-consuming! For me it looked something like this:

ARG POI_V = 4.0.1
RUN rm -f $ TOMCAT_DIR/webapps/alfresco/WEB-INF/lib/poi-$ {POI_V}.Jar
RUN rm -f $ TOMCAT_DIR/webapps/alfresco/WEB-INF/lib/poi-ooxml-${POI_V}.Jar
RUN rm -f $ TOMCAT_DIR/webapps/alfresco/WEB-INF/lib/poi-scratchpad-${POI_V}.Jar

ARG TIKA_V = 1.21-20190624
RUN rm -f $ TOMCAT_DIR/webapps/alfresco/WEB-INF/lib/tika-core-${TIKA_V}-alfresco-patched.Jar
RUN rm -f $ TOMCAT_DIR/webapps/alfresco/WEB-INF/lib/tika-parsers-${TIKA_V}-alfresco-patched.ja

The Amp / Jar built with the 6.2 profile can then be used for testing. But be careful for the 5.2 deployment, the Amp / Jar must be used without the 6.2 profile, otherwise there can be other dependency problems!

Summary

Amps and Jars are still the most popular method of customizing ACS and Share. Testing them locally does not have to be time consuming when using Docker. I hope I have encouraged you to give Docker a try for your extensions in the future and share your experiences with me.

Kudos

Again to Eddie May the new awesome Digital Community Manager from Alfresco for helping improving my english :).

Become a Patreon!

My Smart Home Experience

Fri, 06 Mar 2020 00:00:00 GMT

Hi folks.

As in October 2019 I had the amazing opportunity to buy a house in the beautiful town called Ludwigslust in Germany. It is an old but recently renovated lovely house. As I'm much interested in tech I couldn't wait to try some cool Smart Home stuff.

I made some amazing times with the following Smart Home gadgets:

Smart Bulbs Teckin: That are smart bulbs with integrated wifi. You can easily replace your existing ones with them.

Smart Sockets Teckin: Pretty nice power sockets which you can plug between your electrical device and the socket in your wall to remote control it. It works with your wifi as well.

Smart Garage Door: A cool garage engine extension. It enhances your existing garage engine to be remote controllable via wifi.

Thermostats: Very nice little gadget to time schedule your heaters. Those I tested didn't had wifi or bluetooth connection.

Further done I have links to those gadgets if you would like to try them out yourself. My highest priorities were to save costs like electricity or gas in the long run and increase the living quality from me and my wife.

There are dozens of Smart Home solutions out there and you need to be careful which one you take as often gadgets from different vendors are not playing nicely together.

I decided to buy gadgets which can be controlled with the 'Smart Life' app, as it has a huge offer on all kinds of Smart Home devices.

Which Wireless Frequency?

It is important to mention that all of my Smart Home gadgets only are working in a 2.4 GHz wireless network. It is unlikely that you find gadgets for the 5 GHz network. 2.4 as the length of the waves are longer compared to 5 GHz it reaches better through walls but it has a smaller maximum bandwidth. Actually Smart Home devices don't need a high bandwidth, so thats fine. Make sure to establish both of those network frequencies on your router!

Smart Light Bulbs Everywhere

I opted in buying the Smart wifi LED lights from Teckin. You can buy them in a bundle of for for saving a bit. What can I say, I really love those Smart bulbs. I already saved money with them just replacing the existing bulbs which ran on 40 W to use those Smart bulbs which just need 8 W to shine.

As it is just natural to forget turning out the light and it already happened, I configured every replaced bulb to turn off at 1 am as usually no one is wake at this time. Anyway there are some places in the house, where the wifi signal is not strong enough. For that case I didn't installed a smart bulb. I only replaced the 40 W against a simple LED ones.

Smart Sockets

The next smart device were the Smart Sockets. I love those sockets as they have a bunch of nice features. First as we in Germany don't have manual on off switches integrated in the power sockets like I know from UK, it is very convenient that those smart sockets have an on off switch. Than you don't need to force pulling out if you want to cancel the power and your device doesn't have an on off switch itself.

The next cool feature of those sockets is that you can configure a time schedule for switching off and on devices. I do use a schedule off switch on my TV station as there are a bunch of energy waster which I would like to cut off at night. So I configured the TV smart socket to turn off at 1 am. An other off schedule is on the microwave. As it is an old microwave it spends a lot of energy even just being in standby. That isn't acceptable so the microwave turns off at 7 pm and 1 am now. If you want to turn it on again, simply press the button on the socket.

Last but not least the washing machine. It now has a smart socket between as well. Three reasons for that. As well the on off switch is practical. It saves energy because it turns off the machine completely. Really cool I configured the "Smart Life" app to send me a notification when the cloth is done washing. That is possible because you can trigger a notification event when the power level reaches a threshold . I love that!

Smart Garage Door

Imagine if you could open your door with just saying "Hey Siri, Garage". Well thats the case for me. And still until now I feel like Batman when his cave opens kind of automatically when he approaches in his batmobile :D. So very very cool. How I did that? When I bought the house, there was already an electrical garage door which could be controlled with a simple infrared remote. Yes thats nice but it isn't handy to carry that remote with you all the time and it can't be voice controlled at all. So I wrote down the name of the door engine and bought on Amazon for a Smart Garage gadget which is compatible with that engine and the 'Smart Life' app.

The installation of the device was easy. I have the following enhancements. The old-fashioned remote control is still working but now I can open and close the door with my smart phone and the app 'Smart Life' or using Siri. Furthermore I can check if the garage is open or closed on my phone and I do get an alarm when the door gets opened. And really cool I scheduled the power of the garage engine to turn off the electricity at night. That saves costs and increases the security.

Time Scheduled Thermostats

I am very happy with my "Smart" thermostats from Eqiva Radiator for most of my heaters. Though I am not sure if they are falling under the category of Smart Home devices as they are basically just configurable through the display and I am currently only using them for time scheduled heating and easily turning on and off. Anyway there were very easy to install and they do work like a charm.

Smart Home Gadgets

If you found something I mentioned here interesting, check it out those Affiliates links (I would get a little commission for supporting my blog post work):

Smart Bulbs Teckin (Amazon): UK DE USA

Smart Sockets Teckin (Amazon): UK DE USA

Smart Garage Door (Amazon): UK DE USA

Thermostats (Amazon): UK DE

Future Ideas

I feel like that I just started my little Smart Home journey. Soon I would like to try more complex technologies like the Tado Smart Radiator system. As well I am thinking about to create my own private Smart Home Server running an a PI. I found some interesting articles regarding that in a german magazine.

Summary

Smart Home devices are not just tools for playing around with. They actually help you to save real coins which will add up over the years and will increase your living quality. If you think about peeking in the Smart Home devices world start with simple devices like those wifi Smart Bulbs and Smart Sockets which doesn't need any other devices to work and they are cheap! Be creative for how you can use them in a smart way. And yeah most important have fun!

Become a Patreon!

Build an ADF App from Scratch

Sun, 01 Mar 2020 00:00:00 GMT

Good day

As mentioned in the previous Blog Post, I am working on an exciting AI prototype which uses ACS Community as a content management system. In addition, certain AI services are running and performing certain AI actions using the ACS interface such as CMIS and REST. Unfortunately, I have to remain unclear here to protect company secrets. So I am allowed to say OBJECT and the partner company are planning a webinar where, among other things, this prototype will be shown. Back to the topic! We chose ADF for the webapp.

In the next chapters I will explain how to create an ADF webapp project and how to change it. That shouldn't differ whether you are working with Windows 10, MacOS or Linux. I myself like all three operating systems.

This is Required

You should install Docker, Docker Compose, NPM and YARN. I needed YARN because I am working on a Windows laptop and YARN is able to convert the Windows Paths into Unix Paths. Watch out at Docker! The standard installation at Docker only allows the use of 2 GB RAM for the Docker Compose Deployments, which is far too little for ACS Community and the other services! For this you need at least 10 GB and for ACS Enterprise at least 12 GB. Preferably more! In addition, if you want to set up your own amps, jars or other customizations for ACS deployment, I recommend installing Java and Maven.

Prepare the Git repository

I use the ACA Git Repository as the base for the prototype. Firstly it has already integrated many ADF Component and API Libaries and secondly it provides a shell that uses the ACS backend quite extensively. In addition, it will be easy to import updates from the ACA repository into my webapp. And pretty cool, I can directly contribute back.

Now lets start with the Git repository. For me, the choice was a private Git repository in our OBJECT GitLab. In your case, maybe you can take a public Git repository. The further instructions will not change regarding of choosing a private or public ones.

As mentioned earlier, I would like to be able to import updates from the ACA repository without any problems. In addition, of course, I want to reuse as much of the ACA repository as possible. I therefore chose Git's submodules. With that I am able to contribute back easily. I created a fork from the ACA repo https://github.com/mmuller88/alfresco-content-app. I advise you to do the same. Then simply switch to the project directory and load the fork as submodule:

git submodule add https://github.com/<USER>/alfresco-content-app

The following command may also be useful to update the submodules, which in this case is only ACA:

git submodule update --init --remote

I needed it because I changed the branch in the .gitmodules file to apply my fix branch.

Start ACA

We can test if the Docker Compose Deployment starts in the alfresco-content-app folder. In my previous position as a full stack developer at Alfresco I had the opportunity to work a lot with this ACA repository.

The following commands are required to start ACA Deployment:

yarn install

To load the necessary node dependencies.

yarn run build

For creating the ADF webapp in the dist/app directory. The command also creates the ADF extensions with the names @alfresco/aca-shared and @alfresco/adf-office-services-ext. As an early objective, the goal will be to write your own extension for your own ADF webapp. This has practical reasons such as modularization and distribution.

The start.sh in the ACA directory is a sophisticated script. You should preferably use it to start the deployment. The individual parameters can be queried with the -h or --help flag. For my Windows laptop I have to use these parameter configuration:

.\start.sh -wp -hi 192.168.0.237

The -wp flag converts the Linux file paths to Windows file paths and -hi stands for Host IP and transfers the IP of the computer to the deployment. If everything is all right the following addresses should be available:

http://localhost:8080/alfresco/   ACS
http://localhost:8080/content-app/   ACA Webapp
http://localhost:8080/share/   Good old Share

When I started writing this article I got a strange error because of the GoogleDocs module on my Windows PC. I'm sure I didn't get it on my MacBook. I confidently ignore this error. If you dear reader, however, knows why, I would be happy to receive a private message with an explanation.

To cleanly shutdown the deployment simply use the following command:

.\start.sh -d

The stop and delete all containers that were started using start.sh. You will need to delete and start the deployment again to apply changes from the dist/app webapp. Alternatively you can use the -aca flag for the start.sh script as that only redeploys the webapp container.

Adjust the Deployment

This and the next section is probably the most difficult and requires a lot of patience. So far it was easy because we just took existing code that was already configured and tested by other developers. Now we have to do it ourselves, because we want to be able to integrate customizations like our own extension. So, grab a coffee, or like in my case a tea and get ready!

To manipulate the Docker Compose Deployment, I copied all direct Docker related files from the ACA folder into the project folder. This includes, for example, docker-compose.yaml, start.sh, Dockerfile and the docker folder. It would now be advisable to test whether your Docker Compose Deployment is still working. Maybe just copy the ADF webapp in the alfresco-content-app/dist/app folder into the project dist/app folder and start the deployment with start.sh script. I described how to do this in the previous chapter.

Now comes the most difficult part, at least that's how it was for me. You now have to copy the Angular related files into the project folder and configure them so that they use the components from alfresco-conten-app/src. In my case it was the following files:

src\app\extensions.module.ts
src\assets\app.extensions.json
src\app.config.json
src\tsconfig.app.json

I don't want to pretend here, the correct setting of the configuration actually took me two days. When I got stuck, I looked for advice from the wonderful ADF Community in Gitter. The boys and girls understand their craft. Special thanks to my friend and ex-colleague Bogdan.

Now it is time to let Angular compile the dist/app folder yourself instead of just copying it from the aca project. If you have managed to make Docker Compose Deployment successful with the web app compiled in the parent folder in the dist folder, you can finally start customizing :)!

Create an Extension

Wohoo! You actually made it this far. To build a new extension, the following command is recommended:

ng generate library my-ext

I would highly recommend adding -ext to the extension name to make it more distinguishable from the webapp itself. In my project they have the same names, except that I added -ext to the extension. What you do now with the extension is totally open and there are many great Angular tutorials, pages or forums out there to help you.

If you want to test the webapp in between or at the end of your changes, you simply have to compile the ADF webapp and start the Docker Compose Deployment. This looks similar to what is explained in Start ACA section with the following commands:

yarn install
yarn run build
.\start.sh -wp -hi 192.168.0.237

Please don't forget that -wp is Windows specific. If you use MacOS or Linux you don't need this flag. You probably wouldn't even need -hi IP then.

New ACA Update Available

As the ACA Git Repository evolves at an incredibly fast rate, the question arises how we can integrate these updates into our ADF web app in just a few steps. So far I have not had to do that but basically these steps are mentioned:

The alfresco-content-app Git submodule must be updated.
The dependencies of the package.json file in the project folder must be synchronized with those in the alfresco-content-app/package.json

Summary

Wow think about what we did here. We actually wrote our own ADF webapp which is based on ACA and can be easily updated at any time! For that we first had to create a new Git repo. Then integrate the ACA project as a submodule. Immediately afterwards we tested the Docker Compose Deployment in ACA. Then we rebuilt the repo so that we integrated our own Angular Extension into the ADF web app. That’s it! I hope you had fun and the article was helpful.

Kudos

To Eddie May the new awesome Digital Community Manager from Alfresco for doing some English quality check and offering more feedback for coming posts :).

Become a Patreon!

My First Exciting Week at OBJECT

Thu, 20 Feb 2020 00:00:00 GMT

Hello.

In this article I briefly summarize the first week at OBJECT. For myself that has a couple of advantages. On one hand I practice writing articles and on the other hand it serves as a reflection on the previous days. As well for some of my family and friends it is hart to understand what I am doing in my dream profession as a Software Engineer. This kind of blog post may helps them to understand it better.

Day 1 Onboarding

It was originally planed that I should come to Zurich for onboarding. In short, onboarding usually means the company of the new employee makes you familiar with the hardware and the various software systems they are using, as well as the tasks you have to work on. At that time my wife was pregnant and I didn't wanted to let her alone.

I was very happy when OBJECT offered me to do the onboarding in Hamburg. I love Hamburg because it is a beautiful city surrounded by water. In the article picture you can see the Church of St. Jacobi in Hamburg, which I noticed immediately on the way to the onboarding location. I was given three exciting main tasks. I can only write about these in very rough details.

The first task is that I have my own project together with another partner. This project is about an exciting Alfresco deployment. In this deployment, ACS (Alfresco Content Service) and APS (Alfresco Process Service) should run together. It is super exciting that I am allowed to combine Alfresco's ECM and BPM products.

The second project is about developing a prototype for discovering sensitive data. For that we also would like to create a beautiful ADF app. In this project, my own knowledge gained with Docker and ADF will be very useful.

In the third and for me most exciting project it is about that OBJECT would like to be more in the social media world. That would go from platforms like Xing, LinkedIN to YouTube videos. Again, I have a great interest based on my knowledge of Alfresco and my existing social media skills, which can be improved though.

For this day I can say that it was challenging, but also very productive and great. I am very happy that OBJECT has big plans for me.

Day 2 Project Launch

My great Brazilian colleague, also called Amigo by me, took the time to give me an overview about the current ACS 5.2 deployment. I devoured all the details and asked him a lot of questions. It took us about two hours. The most impressive fact was that the designing process for this deployment. I already have a few interesting ideas on how to improve the deployment in terms of maintainability, security and reliability.

Day 3 Developer Meeting

On Wednesday, and thus the third day, we had an approximately one hour meeting with all developers at OBJECT. In this round all of my developer colleagues summarized in short on what exciting tasks they are working on. In order to better digest this flood on information, I decided to write it down in my notepad. The introduction round was very nice and I think I left a first good impression of myself. I hope that I can soon work together with one or the other colleague because I feel I can learn a lot from them.

I spent the rest of the day to learn more about the deployment I mentioned in the chapter before. I also made ready an improvement proposal for that deployment and I am excited to hear what they will think of it.

Day 4 Inspirational Conversation

Today on Thursday I had a super interesting and inspiring conversation with one of our partners. This is in the area of AI (Artificial Intelligence) and the partner founded that company. Alfresco's Content Management System is just one of many platforms where that AI can extract valuable information. I was impressed as the founder told about the product with so much passion.

My goal is to develop a prototype based on ADF in the next few weeks, which the using the AI capabilities I spoke of. I will use ADF for the prototype. As mentioned in the Blog Post I love this framework. My idea is to include Alfresco's ADF app with the name ACA as the base for the prototype. It might could be a cool idea to write an Artikel about that app too.

Day 5 AI Magic

When I started writing this article, I certainly didn't think that AI would become such an important part of this post. I am very interested in AI and today on Friday I got a cool life demonstration about an AI system that uses Alfresco's Content Management System as a data source to extract extremely interesting information. The system also warns the user if something suspicious happens.

I find it impressive that a simple ACS Community version without any customization was sufficient. This demonstration gave me a good picture of how the ADF app should look like. On Monday I will start making small sketches of the prototype.

Summary

The first week was intense with lots of new information. I was positively surprised that quite a lot colleagues seek my help regarding Alfresco stuff. I see very ambitioned and forward looking into the future from myself together with OBJECT.

Become a Patreon!

From Alfresco to OBJECT

Sun, 09 Feb 2020 00:00:00 GMT

Dear reader.

I am happy to announce that I will switch to OBJECT on February 14th. This is a partner company of my current company Alfresco. OBJECT mainly operates in the German and Swiss ECM (Enterprise Content Management) market. Below I list my motives and at the end I explain my future visions of OBJECT and Alfresco for the year 2022.

Reasons for the Change

One of my biggest motivations is that I really like the Alfresco products. Alfresco Content System (ACS) is an genius and sophisticated content management system based on the Java VM. Even though I hardly had the opportunity to work with Alfresco Process Service (APS) so far, I recognize how great this product is and would like to work with it in the future. Last but not least, Alfresco Developer Framework (ADF). I find the new, Angular-based web interfaces very nice and glorios for the front end. Above all, the simple use of the graphical user interface and the impressive collection of ready-made components, such as the Document List affects me. In 2019 I had the opportunity to help the ADF teams with Alfresco backend know-how and was able to learn a lot about ADF, for which I am very grateful.

The switch to OBJECT should also enable me to be closer to the customer. I want to develop solutions for and with the customer. I am convinced that Alfresco offers excellent solutions in the ECM and BPM area. In cooperation with the customer I will help to find the ideal solution. I would not shy away from considering other software products for solutions or partial solutions besides Alfresco. In fact, I love getting to know new and other software technologies.

I have also discovered the possibilities of the cloud for myself in recent years and was simply overwhelmed. With such a simplicity with which infrastructure such as virtual servers or storage can be provided and operated inexpensively, it is second to none. Acting in the cloud was so much fun that I decided to take AWS certificates. In addition to other previous certificates, I obtained the "AWS Solution Architect Professional" certificate in July 2019, which I am very proud of. If you would like to know more about the certificate and the preparations for it, have a look at my blog post SAP-C01. Therefore, I would be happy if I can promote the use of the cloud in German-speaking countries.

I am also very interested in new IT technologies such as Docker and Kubernetes. I think working with Docker is so nice, because you can always provide clean environments and configure them with just a few lines of code. It is also simply incomparable how much easier it is to develop with Alfresco products if they are already defined in containers. Alfresco also offers reference images e.g. acs-packaging as well as deployments e.g. acs-deployment that are easy to use and customizable. I also don't want to leave out Kubernetes at this point. Kubernetes has a steep learning curve, but when used in production it is simply unbeatable. Alfresco also has a reference deployment alfresco-dbp-deployment. I would be happy to further advance these technologies in German-speaking countries.

Last but not least. The application process with OBJECT was a wonderful experience and really stood out from all of my applications before. The employees at OBJECT make a very happy and open-minded impression. I also think OBJECT's home office culture is great and contemporary. This is the ideal environment to develop myself further.

In the next section I would like to give you an outlook on OBJECT together with Alfresco for 2022.

Outlook 2022

In the last year at Alfresco, I was able to observe the cloud trend and participate in it myself. I know that this will be expanded even more. Alfresco already offers a strong PaaS (Program as a Service) offering and will continue to expand it in the future. Realistically speaking, I believe that this offer will not be used in German-speaking countries until 2022, or will only be used to a limited extent, since countries like Germany are moving very carefully and therefore slowly in the cloud world.

Even if the German-speaking region will hardly use the Alfresco's PaaS offer by 2022, we will experience a strong move towards the cloud with regard to Alfresco products. Already today many costs can be saved by using the cloud as storage e.g. the ACS S3 or Azure Connector makes it easy to use the BLOB storages of the respective cloud provider. OBJECT can provide advice and action as a trustworthy partner.

Many Alfresco customers are already using the chic web applications created with ADF, and there will be more by 2022. I see OBJECT in the perfect position to develop customized ADF apps together with the customer.

Become a Patreon!

Water Fasting Experience

Wed, 12 Jun 2019 00:00:00 GMT

Water Fasting what is that?

As the name describes, you are only allowed to drink water for the time of the fasting. No teas, coffee, sugar-less or sugar-free drinks except, of course water.

As from personal experience it is much easier than juice fasting where you only drink fresh pressed juice from a big variety of vegetables like apples, oranges and cucumbers and so on.

Motivation?

You might think why on earth should someone should do something like this? Well there are some interesting health benefits when we abstain food for a while. And it makes sense when you think where we humans are coming from. Now the days food is every where at any time. Just some decades ago, food wasn't available everywhere all the time. Humans had to survive with less or even no food for days. An unpleasant relic from those time is the capability to store extra calories as body fat.

So what are the health benefits I spoke about? There are a lot benefits and most of them are having scientific evidences. I will focus on my top three reasons to avoid eating for some days.

First: Autophagy. A phase from the body where it self-eats its own body cells to cover the need of protein. Preferable the body starts to eat degenerated body cells first. Those degenerated cells otherwise could cause cancer in the worst case. A better explanation from a study 2010:

[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2990190]

Autophagy also plays a housekeeping role in removing misfolded or aggregated proteins, clearing damaged organelles, such as mitochondria, endoplasmic reticulum and peroxisomes, as well as eliminating intracellular pathogens. Thus, autophagy is generally thought of as a survival mechanism.

So I really hope doing fasting once or twice a year to reduce the amount of those "bad" proteins to have a reduced likelihood for having certain diseases like cancer.

Second: Reset of gut bacteria. As a known fact our gut bacteria are very important for digestion and our immune system. But alongside the settlement of "good" gut bacteria we also can have less useful or even harmful gut bacteria. Those settlements normally following the rule first come, first served. Having a "bad" composition of gut bacteria can reduce not even your health, as well your live quality. A medical article describes it as the following:

[https://www.tandfonline.com/doi/full/10.1080/13102818.2018.1481350]

Gut microbiota dysbiosis may lead to a number of diseases, including gastrointestinal disorders, obesity, cardiovascular diseases, allergy and central nervous system-related diseases.

As all kind of bacteria, gut bacteria need food as source of energy to survive. Taking away food will reduce the total number of bacteria and allow the settlement of new "good" bacteria cultures after the fasting, assuming that you eat healthy food after the fasting period like vegetables or natural yoghurt.

Third: Food detaching. Less of physical health benefit that is more like a mental health benefit. In our modern time, food plays a big role in our life. Some of us are spending much time thinking and consuming food. Experiencing that you can survive without frequently eating, makes you less attached to food by becoming more independent of it.

Fasting experience

Just some years ago in March 2017 I tried my first fasting experience. I decided to do juice fasting for 7 days. For this I bought a juicer which separates the juice from the pulp and some big glass bottles. Every day I drank two bottles full of juice composed of fresh apples, oranges, cucumbers, beetroot, grapes and other vegetables. I had less side effects. What I remember of, I had some minor headache after the third day which disappeared shortly after they appeared. But more interestingly it became very hard, after the fourth day, to continue with drinking the juice because it started to taste very awful.

In the next year 2018 I tried water fasting two times. Both times I skipped eating for three days. I only did three days because I tried to minimize the effect of loosing muscles during the fasting, but receive the full health benefits as described above. I did the water fasting on weekends to minimize the chance to affect my work as a software engineer.

3 Days Water Fasting Diary

In this section I will summarize my three days of water fasting including the day before as day 0 and the day after as day 4.

Day 0 Thursday:

I call it day 0 because it is the last day of eating. Knowing that I will not eat for the next three days I ate plenty of food until 9pm. That helped me to suppress my hunger the following first water fasting day.

Day 1 Friday:

I woke up and drunk two big glasses full of water before I started my 30 minutes morning workout, which I do usually every morning at 8 am. The workout includes 8 minutes tabata. If you don't know what is tabata I explain it shortly. Tabata is a high intensive short workout which usually consist of at least 9 cycles where you perform at least 20 seconds a full body exercise like jumping jack or burpees. After the 20 seconds you rest for 10 seconds. These you repeat 9 times.

After my daily morning workout I took a shower and measured my body weight with my impedance scale which I listed in the end of this article. It ca measure your body composition of body fat, muscle and water as well. In day three I will reveal how much kilos I finally lost. Please keep in mind that loosing weight is not my top priority for doing water fasting, as it is a much better and more secure to loose slowly weight over many weeks and month! But if you know what you do, it can be a nice short cut for losing some kilos.

Additional to measuring my weight and body composition I was interested how fast my body can switch to mainly use the fat instead of sugar for its metabolism and produce ketones for compensating the missing sugar. There is a lot of information out about ketones and why they are healthy in many aspects. For finding out if my body produces ketones I used a simple alcohol breathalyzer. Cheap breathalyzer can not distinguish between alcohol or acetones in your breath, which makes them easy to use and perfect for figuring out if you are in ketose or not. This article (https://www.fatforweightloss.com.au/measure-breath-ketones/) describes more in details about using a breathalyzer for measuring ketones. The following picture is taken from my alcohol tester which shows that I did not produce ketose so far.

[picture Alcohol-tester Friday morning]

I went to work. In the office I paid much attention to drink a lot water. Even it was Friday it was quite a busy day with a lot of meetings. Anyway the work day went on very smoothly and I felt no hunger. Even at home in the night the craving for eating something remained low. Remember I ate plenty food the day before which seemed to pay off! As well before I went to sleep I measured my ketose in my breath.

[picture Alcohol-tester Friday evening]

Day 2 Saturday:

Hello lovely weekend! Day two was slightly more harder regarding hunger but overall the feeling remained low. As usually I made my 30 minutes morning workout at 8 am and after it I measure my weight again. I already lost almost 4 kilos in just on day which surprised me quite a lot. Though I know most of it is water which you are loosing as well when your body is burning the sugar reserves in your body. Even more excited I was about what the alcohol tester is saying and if I am in ketosis or not.

[picture Alcohol-tester Saturday morning]

A value over 0.030 would indicate that I am in ketosis and my body mainly burns fat to produce energy.

It was a sunny day and most of the rest of the day my wife and me we spend outside in the park enjoying the sun. Of course I payed much attention to drink water. The hunger craving remained low, but it made me a bit jealous seeing my wife eating and smelling this delicious food she ate. Once more I measure the ketosis in my breath.

[picture Alcohol-tester Saturday evening]

Day 3 Sunday:

Finally the last day. It was my favorite day, knowing that I can start soo eating again. I followed my normal morning routine. After the workout I measure my weight. I lost unbelievable 6 kg. The alcohol tester reached it maximum:

[picture Alcohol-tester Sunday]

At 1 pm I prepared a natural yoghurt with some healthy nuts in it. Remember one of my health benefits was to resettle good gut bacteria. Those bacteria normally come with healthy food like yoghurt. At 2 pm my wife prepared a healthy warm meal for us.

Day 4 Monday:

For luck this day was a bank holiday in UK. I woke up with a feeling of endless energy which I directly could use in my morning workout. I felt stronger than never before. This is easy explainable. After abstaining carbs for some days your body becomes very sensitive for carbs and tries to store it even stronger in your body.

Summary:

As mentioned before it wasn't my first water fasting and my experiences were similar as the times before. I enjoyed the few days without eating and I really can recommend to try it yourself! In special after the first time you will be amazed for what your body is capable of for abstaining food.

Just some advices from me if you plan to do water fasting as well. Reduce high intensive workouts but keep doing workouts! Do it when you are not working like on Weekend or during holidays. Observe and write down the experiences you made to reflect on them.

Disclaimer: The following section contains affiliate links, which means that if you click on one of the product links, I’ll receive a small commission. This helps support this blog!

My Amazing Equipment:

Alcohol tester: Great for figuring out if you are in ketosis or not.
Amazon UK: https://amzn.to/2IdwHGo (Amazon)

Omron Scale: Measures not only your body weight. Can estimate your body fat, water and muscle composition as well.
Amazon UK: https://amzn.to/2EPEVUc (Amazon)

Bare-Foot Shoes: Perfect shoes for workouts like tabata, gym or running. Normal modern shoes are bad for your feet as they are overprotecting to your foot which will result in underdeveloped foot muscles and skeleton. A good developed foot is the basement of your body and prevents all kind of resulting issues like knee or back pain. There are many of expensive bare-foot shoes out but some cheap water shoes are completely sufficient.
Amazon UK: https://amzn.to/2ESuyiF (Amazon)

Bluetooth Earphones: I tested many different bluetooth earphones. I am sticking now with this one because they are cheap but are from very good quality regarding sound, battery and robustness. I can use them for weeks for my workouts without ever charging them!
Amazon UK: https://amzn.to/2IkCdqG (Amazon)

Gymnastic Rings: Perfect tool for upper body workout. I love it to train outside in the sun and fresh air. As well they are very light so you can take them with you wherever you go.
Amazon UK: https://amzn.to/2WLwKm4 (Amazon)

Stainless Steel Water Bottle: That drinking water is crucial I don't need to explain. I prefer those steel water bottles because there is not risk of plastic in your water.
Amazon UK: https://amzn.to/2JYGyTL (Amazon)

Become a Patreon!

Martin Mueller's Blog

OpenNext + CDK: Next.js on AWS Without ECS Fargate

The product: qr-plakat.de

Why OpenNext + CDK for an MVP

Stack

GitHub Actions — the part I love

Agentic loop

MCPs I actually used

Skills that helped

Would I do it again

Further reading

Sovereign Cloud on AWS: When eu-central-1 Is Enough — and When You Need ESC

Why now?

What ESC actually is — beyond "servers in the EU"

Decision framework: three buckets

Bucket 1 — eu-central-1 is enough

Bucket 2 — ESC for defined workloads

Bucket 3 — Full sovereign stack

What changes for builders

Case studies: who is betting on ESC?

SCHUFA — finance & credit data

Diehl Metering — smart metering & critical infrastructure

University Hospital Essen — sovereign AI in medicine

Medizinische Universität Lausitz – Carl Thiem — regional health ecosystem

ITZBund — federal IT

EWE AG — energy

Eterno Health — health SaaS (hybrid thinking)

Platform partners (not end customers, but relevant)

ESC vs EU hosting on Hetzner

CLOUD Act & third-country access — honestly

Next step: ESC readiness assessment

Further reading

OpenClaw, Three Months Later: What Actually Shipped

Quick recap

1. Blog posts — from idea to publish

2. GitHub issues → plans → PRs

3. Invoices — the boring work I avoided

4. SEO, WordPress, compliance — one prompt, one artifact

5. Telegram topics = project switcher

6. Conference talk drafts

Numbers (rough, three months in)

What's next

Conclusion

Data-Driven SEO with the SISTRIX MCP: A HalloCasa Case Study

What is SISTRIX?

Why MCP for SEO tools?

Setting up the SISTRIX MCP

SISTRIX MCP tool surface

HalloCasa case study: what the data showed

Baseline snapshot

Where rankings actually live

Near-win keywords (fastest ranking upside)

Competitors (DE organic overlap)

Backlinks: mostly noise

Code cross-check: why the directory does not rank

Prioritized roadmap (90-day lens)

How to replicate this workflow

Sample deliverable: HalloCasa SEO strategy (PDF)

Takeaways for other sites

What is next for HalloCasa

Built with Lovable? Host Prod in Germany & DACH on Hetzner

Why “EU is fine” is not enough in DACH

Phased prod on Hetzner — dev stays on Lovable

Open source stack — no black box prod

What I handle vs what you keep doing

Further reading

Production on Hetzner for a German Client — EU Hosting (Part 1)

Why Hetzner, and why not “move everything at once”

What prod looks like (in plain terms)

What we learned (short version)

Hetzner migrations — how I can help

Further reading

Self-Hosted Insights on Hetzner — Observability, Cost, AI Agents (Part 2)

Insights without leaving the EU

Predictable cost, watched automatically

AI agents with real tools

What we learned (short version)

Hetzner migrations — how I can help

Further reading

How I Use OpenClaw as My AI-Powered Personal Operating System