This post covers the general techniques used for this kind of effect. A disclaimer first: I have no knowledge of how any particular commercial product is built. When I mention a product, it is only an example of the kind of output being discussed. Everything below is a general description of public, well-known approaches.

The two families of anime filters

Broadly, there are two kinds of anime effect:

1. Whole-image stylisation. The entire photo is converted into an anime look: flat shading, strong outlines, simplified colour. Classic examples are image-to-image GANs such as AnimeGANv2 (with a popular PyTorch port) and one-shot face stylisation methods like JoJoGAN.
2. Localised, mixed-media edits. Most of the photo stays real, and only one region is changed. The "anime star eye" trend is a good example: one eye becomes a drawn anime eye with a star, heart or manga-ink look, and the rest of the face stays photographic. Anime Star Eyes (animeselfie.com) is one product built around this format. It takes one adult selfie, replaces one eye, adds original "anime twin" character stickers and returns a 2K 9:16 image.

The second family sounds easier, since you only change a small area. In practice, it brings its own set of problems.

Step 1: Find the face

Everything starts with face detection: a bounding box (or several) around faces in the image. Modern detectors are fast and robust, but a production pipeline still has to decide what to do with:

• No face (a pet, a landscape, a heavily cropped shot)
• Multiple faces: which one gets edited?
• Tiny faces in a wide shot, where the eye is only a few pixels wide
• Extreme poses, where one eye is barely visible

Many consumer tools sidestep part of this with input rules: one person, face clearly visible, size and format limits. Those rules are not just bureaucracy. They narrow the input distribution so later stages behave predictably.

Step 2: Localise the eyes precisely

A bounding box is not enough to edit an eye. You need facial landmarks: dense keypoints around the eyelids, iris, brows and so on. Libraries such as Google's MediaPipe Face Landmarker produce hundreds of 3D landmarks and head-pose estimates in real time, which gives you:

• The eye contour (to build a mask)
• The iris centre (to place a pupil or highlight)
• Head yaw, pitch and roll (to warp a flat anime eye so it matches the face's angle)

Landmarks get unreliable with glasses (reflections and frames), hair over the eye, heavy makeup that changes the apparent lid line, and blinking or squinting. This is why almost every guide for these filters tells you to take off glasses and keep the eyes clear.

Step 3: Decide how to change the eye

There are roughly three strategies, each with trade-offs.

A. Template compositing. Take a pre-drawn anime eye asset, warp it to the landmark geometry, and blend it in. This is predictable and fast, but it can look "stickered on" because the lighting, colour and scale never quite match the photo.

B. Global style transfer, then masking. Run a stylisation model on the whole face (see neural style transfer for the general idea), then keep only the eye region using a mask. It is easy to build, but whole-image models are trained to stylise everything consistently. Cut out a single region, and the seam between styled and unstyled pixels is often obvious.

C. Masked generative inpainting. Mask the eye region and ask a conditional generative model to fill it with an anime-style eye that fits the surrounding pixels. This can produce the most natural blends, but it is harder to control: the model may drift on identity, change the wrong eye, or produce an eye that does not match the requested motif (star versus heart).

Real systems often mix these: generative fill for the blend, plus constraints or references to keep the motif consistent.

Why a single-eye edit is genuinely hard

Changing one eye breaks several assumptions that whole-image filters rely on.

1. Symmetry is now a bug, not a feature

Face models learn that left and right eyes are similar. If you ask a generative model to stylise one eye, it often "leaks" the style into the other eye, or pulls the edited eye back towards realism. You need a tight mask and strong conditioning to keep the asymmetry.

2. Two rendering styles have to meet on one face

The seam between a photographic eyelid and a flat, ink-outlined anime eye sits right on a high-attention area. People are extremely sensitive to anything odd around eyes. Feathering the mask helps, but too much feathering produces a muddy halo. Too little, and you get a hard cut-out edge. Some looks, such as manga screentone, deliberately keep a graphic edge, which is a design choice that hides the seam by owning it.

3. Scale and perspective

Anime eyes are stylistically oversized. The edit has to be bigger than the real eye without covering the brow or nose in a weird way, and it has to follow the head's rotation. A frontal eye pasted onto a three-quarter face looks wrong immediately.

4. Lighting consistency

The real half of the face has a specific light direction and colour temperature. A convincing anime eye needs highlights that roughly agree with it, even though anime highlights are stylised (stars, hearts, sparkles).

5. Identity preservation

The whole appeal of the mixed-media look is that it is still recognisably you. Any generative step that touches the face risks subtly changing the nose, skin or expression. Keeping edits local and the mask tight is the main defence.

Extras: generated characters and output format

Some products go beyond the eye. Generating a separate chibi-style "twin" character from the photo is a different task: it is closer to reference-guided character generation than editing, using cues like hair colour, outfit and accessories. Layout then becomes a composition problem: place stickers without covering the face.

Output format matters too. Social apps are vertical, so producing a fixed 9:16 frame at decent resolution (for example 2K) avoids a lossy crop and re-upload later.

Product-level concerns makers should not skip

If you are building something like this yourself:

• Restrict inputs (one person, clear face, sane file sizes and formats) to cut failure cases.
• Treat face images as sensitive data. Private by default should be the baseline.
• Add safety checks and age gating. Face-editing tools are easy to misuse on other people's photos. Animeselfie.com, for example, states its service is 18+ only.
• Handle failures gracefully. Detection and generation will sometimes fail, so decide what the user experience and billing look like when they do.

Wrapping up

"Change one eye to an anime eye" combines face detection, landmark localisation, pose-aware warping, masked generation and careful blending, all aimed at the region of the face humans scrutinise most. If you want to experiment, the open-source projects linked above are good starting points. If you want to see what a finished, single-purpose version of the idea looks like, an AI star, heart and manga eye selfie tool like AnimeSelfie is a useful reference for the target output.