How We Built It, Part 10: Putting a CRM Inside Your Website

Part 9 ended with the maintenance plan. This one is about growth: what happens when the site needs to do something a CMS does not do — in our case, manage members. Contacts, memberships, payments, event registrations. That is a CRM, and CiviCRM is the open-source one built for exactly this.

You have two ways to run it. Both work. Only one of them does what people actually want.

The decision that looks technical and isn't

CiviCRM can run standalone — its own site, its own login, its own address. That is the fastest thing to stand up, and it is what we had: a working instance with 463 real contacts on its own subdomain.

Or it can run inside your CMS, sharing one codebase, where a site user and a CRM contact are the same record.

The difference only becomes obvious when you write down what a member is supposed to be able to do. Sign in once. Manage their membership. Pay their dues. Register for an event. Sign up as a sponsor. Every one of those is a sentence about the website, not about a separate admin tool. Under standalone, each becomes "and now log in again, somewhere else, with different credentials."

So it is not really an architecture question. It is a question about whether you read what people asked for. We had this written down for months and still managed to present it back as an open A/B choice, which is its own lesson: when a requirement is stated as an assumption rather than a question, it has already been decided. Ours was stated three separate times — "extensions for Drupal", "CRM tasks via the website", "integration with Drupal" — and we still asked.

There is one genuine cost, and it should be said out loud rather than buried: the CRM now constrains which version of your CMS you can run. That is inseparable from the benefit. Users being contacts requires one codebase.

Declaring it, without breaking production

Our site installs everything through composer during deploy, and the same script runs in staging and production. That is normally a feature. Here it is a hazard: adding a package means the next production deploy installs it too.

So the whole thing sits behind a flag, and the flag is the only thing keeping it off the live site:

if [ "${CIVICRM_ENABLED:-0}" = "1" ]; then
  composer require --no-interaction --no-update \
    civicrm/civicrm-core:~6.16.0 \
    civicrm/civicrm-packages:~6.16.0 \
    civicrm/civicrm-drupal-8:~6.16.0
fi

Staging sets it to 1, production to 0. Bound to a series (~6.16.0), never an exact version, so security releases still arrive on their own — pinning contrib to an exact version is how you keep a known-vulnerable module forever.

Two notes if you copy this. civicrm-drupal-8 is the right package for Drupal 9, 10 and 11 despite the name. And a big application like this cannot be installed by a Drupal recipe — it writes its own settings file, creates ~158 tables in a separate database, and runs a compile step. Ours installs from a deploy script that checks whether the schema exists and either installs or upgrades. That self-gating is what lets the site be rebuilt from scratch without wiping the CRM.

Ten things that broke, and what they teach

The install itself took minutes. Everything below is what actually consumed the day — and almost none of it is specific to CiviCRM. This is the transferable part.

1. Dry-run before you touch a shared path

The first attempt never ran. A dry-run resolve found that CiviCRM requires cweagans/composer-patches ~1.0 while our site required ^2.0 — a hard conflict in a file that every future deploy of everything depends on. Thirty seconds of composer update --dry-run in a throwaway copy prevented breaking every unrelated deploy for a week.

If a change touches a path shared by things you are not currently working on, prove it resolves before you commit it.

2. A floating image tag is frozen, not floating

This one is worth the whole post. Docker Swarm resolves an image tag to a digest when the service is created, and never re-resolves it. A later update that does not change the image line keeps the old digest.

So image: something:latest reads as "always current" and means "whatever was current the day this service was first created." Ours had been created in April. In August it was still running the April build — three minor versions behind, and inside the affected range of a critical remote-code-execution advisory — while the dashboard cheerfully displayed a recent "updated" timestamp, because a config-only update had touched the service without re-pulling.

Do not trust the tag string. Compare the running digest against what the registry resolves the tag to now:

docker service inspect <service> --format '{{.Spec.TaskTemplate.ContainerSpec.Image}}'

And when you audit for this, grepping for :latest is not enough. We did exactly that, declared ourselves clean, and missed a :release tag that rolls identically. Some images publish no version tags at all — those must be pinned by digest.

3. Read the error, not the word

A payment extension "wasn't installing due to firewall issues." That had been believed for weeks.

It required a dependency named firewall. The missing-dependency error mentions firewall, which reads exactly like a blocked connection. There was nothing to open, no port, no network problem. There was a second cause underneath: the extension needed a newer core than was installed, so it would have refused anyway.

When a diagnosis has been sitting unchallenged, re-run it yourself before you act on it — especially before you ask someone else to change infrastructure for you.

It installed on the first attempt once the two real causes were addressed: the dependency named, and the core version high enough. Total infrastructure changes required: none. What it actually needed was for someone to read the error text instead of the word inside it.

4. A branch deploy that only half-applies

We pointed the staging stack at a feature branch and deployed. The build reported success. Nothing changed.

The stack's git reference selects the compose file. Our init container does its own git clone to populate scripts and config, and without --branch it always took the repository default. Result: the branch's compose file, the main branch's scripts. A half-deploy that fails silently and looks like your change did nothing.

The tell was precise and worth remembering: a log line we knew the new script printed — including its else branch — never appeared at all. Not the wrong branch. Neither branch. That means the file never ran.

5. Config on a persisted volume outlives your script

A failed run had already written a bad dependency into composer.json, which lives on a volume that survives deploys. Removing the line from the deploy script did nothing, because the script was not what was failing — the file was. Every subsequent deploy re-failed on a line no longer in any source file.

Deploy scripts that write into persisted state need to be able to un-write it, not merely stop writing it.

6. Tools that try to prompt

Two separate aborts came from this. Composer refuses to run un-allowed plugins and stops the whole update; CiviCRM ships three. Then its compile step tried to ask a question — "Cannot prompt for compilation preferences" — which is fatal in a container with no terminal.

Anything that runs unattended needs every interactive decision pre-answered.

7. A privileged helper writing files the web server cannot read

The most instructive failure of the day, because every check said it was fine.

Container exec runs as root. The web server runs as www-data. Command-line tools run as root leave root-owned cache directories. The web server then cannot write them, and the site returns HTTP 500 — while every command-line check continues to pass, because those run as root and never encounter the permission the browser hits.

It cost a site outage and 835 root-owned files. The fix is to run as the serving user:

su -s /bin/sh www-data -c "HOME=/tmp cv upgrade:db"

This is the concrete reason for a rule we now apply everywhere: verify by rendering the page, not by reading an exit code. A CLI that reports success is not evidence the site works.

8. The fix that broke it a different way

The fix for number 7 was to stop running the tool as root and run it as the web server user instead. That fix was correct, and it broke the tool completely for a day without anyone noticing.

The tool is downloaded during deploy to a temporary file. mktemp creates files with mode 0600 — owner only. The script then made it executable with chmod +x, which turns 0600 into 0700. Still owner only. Root could run it; the web server user could not even read it.

Every invocation failed with Could not open input file, printed between two success messages. The database upgrade never ran. The version probe reported unknown. A ticket was filed blaming the probe.

What let it survive was the safety check meant to catch exactly this:

# Checks that ROOT can run it. Every real call is as www-data.
if ! "$CV" --version; then
  echo "not runnable — skipping"
fi

The check ran as the deploy user, so it passed on a file only the deploy user could read. Verifying that root can do something tells you nothing about whether the web server can. Check as the identity that will actually do the work — the same lesson as number 7, arriving by the opposite route.

Two characters, chmod 0755 instead of chmod +x. On the next deploy the upgrade that had been silently skipped for a day ran, and the payment extension installed on the first attempt.

9. A success message that could never print its failure

This one had been running for weeks, on a shell idiom that looks entirely reasonable:

run-the-thing 2>&1 | tail -5 && echo "Done." || echo "Failed."

In a pipeline, the exit status is the last command's — tail's — and tail essentially always succeeds. So && always fires and || is unreachable. That "Failed." branch had never printed once, in any deploy, ever. It was decoration.

The script it was wrapping had begun crashing partway through, on a change made elsewhere. It crashed on every deploy for weeks. The log said Done. every time. And because it crashed before reaching the section that sets the marketing copy on blog posts, every article on the site quietly served the stock template's sales pitch — "Get ready to transform your operations" — instead of ours.

Nobody noticed, because the pages returned HTTP 200 with correct headings, navigation and footer. The end-to-end test suite was green. It was green because no test asserted that text at all.

Eighteen call sites had the same idiom. Two of them mattered a great deal: one reported two-factor authentication as enabled whether or not it was, and one reported the CRM module as enabled the same way. Fixing them surfaced two more bugs that had been failing silently on every deploy — including number 8 above, whose error message had been printing into the log the whole time, between two green ticks.

10. A protection that was present and inert

The CRM stores uploaded files — member documents, attachments — in a directory under the site's public files path. It ships an .htaccess file in that directory specifically to stop the web server serving them.

We run nginx. nginx ignores .htaccess entirely. The protection was present, correct, and doing absolutely nothing. Anyone who guessed a filename could download it. No login.

We proved it rather than assumed it: put a file in the directory, fetch it in a logged-out browser, get HTTP 200 and the contents back. Then fix, then fetch again, and get 404. That before-and-after is the only evidence worth having, because the first attempt at the fix moved the existing files but did not change where the application writes them — so it recreated the public directory on next use. It looked fixed and was not.

The general lesson is worth more than the specific bug: a security control inherited from a different web server is not a control until you have tested it on yours. Anything a dependency does for you on Apache, verify on nginx.

What you should take away

The pattern under most of these is the same. Something reported success while not having done the thing: a tag that says latest and means April, a deploy that says success and ran the wrong file, a command that says OK because it is root, a phar that cannot be read by the user that needs it, a shell idiom whose failure branch is unreachable, an .htaccess on a server that never reads one. Exit codes are not evidence. Rendered pages and compared digests are evidence.

And a test that has never been seen to fail is not evidence either. Two separate regressions ran live through a green suite here — a banner that vanished from every page, and the blog copy above. In both cases tests existed and were watching the right feature. They just could not fail: one matched a string that also appears in the footer, another matched an element that something else happily replaced. Before trusting a new test, break the thing on purpose and watch it go red.

The other half is about reading. The architecture decision had been made and written down; the "firewall" diagnosis had never been tested; the version pin we quoted as a constraint had been out of date for three weeks. In each case the information was already there and nobody had gone back to it.