Why AI Sometimes Changes Your Face — and How to Stop It

"It looks great, but it does not look like me." This is the single most common piece of feedback about AI portrait generation, and it is rarely mysterious. Identity drift has a small number of causes, most of them under your control. Here they are, ordered by how often they turn out to be the reason.

First, understand the trade-off the model is making

Every generation balances two instructions that partly contradict each other: reproduce this face, and satisfy this theme. The theme wants specific lighting, a specific mood, sometimes a specific era of styling. Your face, as photographed, has its own lighting and mood. Where those conflict, something has to give.

The model resolves this by keeping the features it is most confident about and reconstructing the ones it is least confident about. Anything your source photo showed clearly tends to survive. Anything it showed ambiguously gets invented. This single mechanism explains almost every case of drift.

Cause 1: the source photo was too soft

By a wide margin the most common cause. A face that is slightly out of focus, slightly motion-blurred, or heavily compressed does not give the model confident information about the fine geometry around the eyes and mouth — and that geometry is what makes a face recognisable.

The test: open your source photo and zoom to 100% on the eyes. If the eyelashes are individually visible, you are fine. If they are a smudge, the model is working from a smudge too.

Cause 2: the face was lit from the wrong direction

Light from above or behind hides the shape of the face. Overhead lighting in particular fills the eye sockets with shadow, and the eye region carries more identity than any other part of the face. The model reconstructs what it cannot see, and reconstructed eyes are generic eyes.

The fix is the same as always: front-facing soft light, ideally a window on an overcast day.

Cause 3: the angle was too extreme

Past about thirty degrees off-axis, half the face is inferred rather than observed. Past forty-five, most of the far cheek and the far side of the jaw are guesses. A profile photo is the hardest possible input for identity preservation.

The same applies vertically. A photo taken from below widens the jaw and shortens the forehead; the model reads that as your actual bone structure and builds a face with a wider jaw.

Cause 4: the theme is doing too much

Some drift is not a fault — it is the theme working as designed. A theme that turns you into an oil painting, a comic-book character or a figure in period costume is explicitly asking for stylisation, and stylisation is deviation from photographic accuracy.

If identity preservation is the priority, choose themes that keep photographic realism: studio portrait styles, editorial lighting, seasonal or location themes. Save the heavily stylised ones for when looking exactly like yourself is not the point.

Cause 5: heavy makeup, filters or retouching in the source

If your source photo has already been through a beauty filter, you are asking the model to preserve the identity of a face that has been smoothed, reshaped and enlarged around the eyes. It will faithfully preserve that altered face. Then the theme deviates from it, and the result is two layers of change away from you.

Always start from an unfiltered photo. Skin texture is identity information, and smoothing filters delete it.

Cause 6: your expression fought the theme

A wide open-mouth laugh involves most of the muscles in the lower face. A theme that calls for a calm, composed expression has to reverse all of that, and the reversal is approximate. Neutral or lightly smiling source photos give the model the least work to undo.

Cause 7: you only tried once

Because generation samples randomly, the spread between two runs of the same theme on the same photo can be larger than the spread between two different themes. One bad result is not evidence about the theme. Three bad results is.

A diagnostic order of operations

When a result does not look like you, do these in order and stop as soon as it improves:

1. Regenerate once, unchanged. Costs nothing and fixes a surprising proportion of cases.

2. Swap to a sharper, front-lit, front-facing source photo.

3. Remove filters and heavy retouching from the source.

4. Switch to a less stylised theme.

5. If it still does not look like you, the theme is simply not compatible with your face, and no amount of input tuning will change that. Pick another one.

What you cannot fix

Some things are outside your control. Very distinctive features — unusual hair, a strong scar, a prominent birthmark — are frequently softened or omitted, because the model treats them as noise relative to what it has learned faces normally look like. The same applies to specific jewellery and to tattoos. If those details are part of how people recognise you, generated portraits will always be a slightly abstracted version of you, and that is a limitation of the technology rather than of your input.