FloatShell – Tecker's Universe

Use These 4 Principles to Improve AI Coding

Wed, 27 May 2026 00:00:00 +0000

Andrej Karpathy recently shared a sample CLAUDE.md with only four rules.

Someone tested it: coding accuracy reportedly went from 65% to 94%.

The model did not suddenly become smarter. The constraints became better.

Karpathy needs little introduction: former Director of AI at Tesla and a founding member of OpenAI. What he identified was not “how to make AI write better code” in the abstract, but four common failure modes of AI coding assistants, each paired with a behavioral contract.

Those four rules were later expanded into twelve and earned 82,000 stars on GitHub. But the core idea did not change: the goal is not to make AI “smarter”; it is to stop it from falling in the same places again and again.

Why Your AI Keeps Repeating the Same Mistakes

Open your CLAUDE.md, and you may find lines like these:

Write high-quality code.
Think carefully.
Keep it simple.

And then?

The AI nods, says “sure”, and keeps making the same mistakes.

It starts coding before the requirement is clear. It fixes a small bug and casually changes a neighboring area. It assumes the project follows some convention, then writes an implementation that looks reasonable in isolation but does not fit the current codebase at all.

The worst part is not that it makes mistakes. The worst part is that the mistake-making process can be deceptively plausible. It drifts a little, patches a little, drifts again. You watch it work and may even feel that it is getting close. Only at the end do you realize it was not approaching the answer; it was adding more patches inside a polluted context.

A Real Example

I had an old Go project that managed dependencies through GOPATH, while my local machine had a newer Go version that defaults to Go modules.

Every time I asked an agent to run tests, it executed go test ./... directly. After the error appeared, it began correcting itself: first trying go mod init, then GO111MODULE=off, then manually setting GOPATH to some guessed directory.

It took four or five rounds of self-correction to get there.

In reality, one line in CLAUDE.md would have solved it: “This project uses GOPATH mode for dependency management.”

But the agent did not know that information existed, so it had to rediscover it through trial and error every time.

What I wanted to solve was not “how do I make AI smarter?”

It was a simpler question: can there be one place where we write down recurring project-specific problems, so the agent sees them before it starts working?

Memory Systems Do Not Solve Team Collaboration

At first, I used the memory system built into my IDE. I put project conventions, common pitfalls, and personal preferences into it. It helped, but it was not stable enough.

One problem is that memory is too personal. For solo projects, that may be fine. In a team project, it becomes an invisible fork: the same repository, but different developers’ agents work with different memories. One agent knows the project must not introduce a certain library; another does not. One agent knows an old module must not be casually refactored; another does not. The codebase has not forked, but the context already has.

Another problem is uncertain loading. You think the agent remembers something, but you do not know whether it actually saw that memory in the current task. Simple tasks may be fine. In long, complex tasks, critical rules may be pushed out of context or selectively forgotten.

I eventually realized that truly important project rules should not live only in one person’s private memory. They should live in the repository, be version-controlled, and be visible to everyone together with the code. That is what CLAUDE.md means to me: not a longer system prompt, but a repository-level collaboration agreement.

Stop Overusing `/init`

When I first used Claude Code, I also tried /init. It scans the project and generates a CLAUDE.md, which feels convenient. But I quickly noticed a problem: it loves turning code details into long-term context.

Directory structures, module descriptions, current commands, implementation details: these can help in the short term, but they easily become stale maps. Once the code changes, the description in CLAUDE.md starts to rot. Worse, the agent may trust the stale description instead of reading the code again.

I now prefer to leave code details in the code. AI coding agents are already good at searching on the spot: reading callers, checking exports, inspecting tests, and following helper functions through the local context. The root CLAUDE.md should not take over that job. It only needs to tell the agent what it must know every time, and which boundaries it must not cross.

Later, I tried clearing CLAUDE.md and writing it again from scratch. I kept only two kinds of content: general coding rules for the AI, and project-specific rules. The result was better than expected. The agent still made mistakes, but it wandered less, changed unrelated code less often, and produced code with better style and taste. It was as if it finally understood that this project was not a blank exercise field for improvisation.

Karpathy’s 4 Rules

Karpathy’s four coding rules are short. So short that you may wonder, “that’s it?”

1. Think before coding. State your assumptions. Surface tradeoffs.
   Ask before guessing. Push back when a simpler approach exists.

2. Simplicity first. Minimum code that solves the problem. No
   speculative features. No abstractions for single-use code.

3. Surgical changes. Touch only what is asked. Do not "improve"
   adjacent code, comments, or formatting. Match existing style.

4. Goal-driven execution. Define success criteria. Loop until
   verified. Do not narrate steps; tell me what success looks like.

In practice, they answer four questions.

Rule 1 - Think before coding. Do not start writing immediately.

The most common AI mistake is to stay silent when the requirement is ambiguous and start coding anyway. When the misunderstanding becomes visible, the work has to be torn down and rebuilt.

The contract is: before non-trivial work, state the assumptions. Ask when details are uncertain. If there is a simpler approach, push back and say so.

Rule 2 - Simplicity first. Do not over-engineer.

To look “professional”, AI often adds abstractions, configuration, and helper utilities. It may look impressive, but future agents will keep building on top of that complexity.

The contract is: write the minimum code needed to solve the current problem. Do not implement features the user did not ask for. Do not abstract logic that is used only once.

Rule 3 - Surgical changes. Do not casually refactor nearby code.

In old codebases, ugly code often remains not because nobody sees it, but because nobody can safely move it. It may carry historical compatibility, customer-specific behavior, or patches added after production incidents.

AI has no such historical burden. It sees duplication and wants to remove it. It sees inconsistency and wants to normalize it. In a bug-fix task, that is risk.

The contract is: only change the parts required by the task. Do not “improve” neighboring code, comments, or formatting. Match the existing style.

Rule 4 - Goal-driven execution. Do not monologue.

The most dangerous AI response is often not “I don’t know”, but “I think I know.” A confident wrong answer is more dangerous than an explicit uncertainty.

The contract is: define success criteria. Keep verifying until they are met. Do not narrate steps; tell me what success looks like.

These four rules are not “I hope you behave this way.” They are “in this situation, you must behave this way.”

That difference matters.

Why 4 Rules Can Be Better Than 20

Karpathy’s four rules were later expanded into twelve. But there is a counterintuitive observation: after the rule count exceeded fourteen, reported compliance dropped from 76% to 52%.

Why?

When there are too many rules, each rule becomes lighter. The model averages them away, and eventually none of them remains sharp.

Six rules that fit your project are usually more useful than twenty copied from someone else.

So my current approach is to start with Karpathy’s four rules as a baseline, then add project rules one by one according to real failure modes. Every new rule should map to a specific incident: “last week, the AI fell here.”

When adding a rule, ask yourself: without this rule, in which concrete situation would the AI fail?

If you cannot answer that, do not add it.

From 4 Rules to 12: When to Expand

Karpathy’s four rules address the moment of writing code. But in real agentic workflows, the failure modes go beyond those four.

The later eight additions cover gaps like these:

Token budgets are not suggestions: in long debugging sessions, AI may repeat attempts, pollute context, and drift farther from the problem.
Checkpoint after important steps: in multi-step tasks, errors compound through later steps.
Tests should verify intent, not only behavior: AI may write shallow tests that pass without validating the business intent, creating false confidence.
Surface conflicts instead of averaging them: when two coding styles conflict, AI may average them into a third, stranger style.
Match codebase conventions even when you disagree: AI may introduce a new paradigm it considers better, leaving two patterns in one codebase.
Use models only for judgment calls: letting an LLM handle deterministic logic such as retries, routing, or status codes is wasteful and risky.

These additions do not invalidate the original four. They say: protect the baseline first, then expand only where needed.

My suggestion: use the four rules for two weeks in a new project. Whenever AI fails, ask whether the failure was outside the coverage of those four rules. If yes, add one rule. If not, improve the wording of an existing rule.

Project Rules Should Not Become an Encyclopedia

General rules are only the foundation. What makes an agent work smoothly inside a project is the project’s own rules.

I prioritize writing down things like:

the tech stack and dependency manager
common commands
how to run local validation and tests
external dependencies and middleware

The validation loop is especially worth making explicit. For example:

After changes, run pnpm typecheck first.
For API behavior, run pnpm test:api.
For UI changes, start the local dev server and confirm key pages open in a browser.
If any validation command cannot run, stop and explain what was skipped and why.

There is another category of project rules that I do not try to invent upfront. I add them while using the agent. When a task does not go smoothly, I may ask it to review the failure at the end:

Carefully review the mistakes you just made, and summarize the lessons as no more than two concise project development principles.

You can even turn that sentence into a skill and run it after each task. Then a human can decide whether the lesson deserves to be written into CLAUDE.md. Only project rules that grow out of real failures are likely to help next time. Real engineering has no silver bullet.

The Root CLAUDE.md Is More Like Onboarding

In daily work, I treat an AI coding agent like an intern who has just joined the project. It is smart, reads quickly, and is willing to work, but it does not know why the project looks the way it does, where it must not touch, or where the team has been hurt before.

When onboarding a new teammate, you do not dump the company’s history and every module’s implementation details into their lap. You first explain a few things: what the project does, what stack it uses, how to test changes, which directories are high-risk, and where to look when uncertain.

CLAUDE.md is similar. The root file should contain the minimum knowledge needed every time, not details that frequently go stale. Let the agent search code details on the spot. Put optional context in docs/ or Skills, and let the model load it gradually when relevant.

A directly usable structure can be short:

# CLAUDE.md

## Project
This repository is a [project type].
Core goal: [what the system is for].
Main users: [internal operations / customers / developers].

## Behavior Rules
- Think before coding: before non-trivial work, restate the requirement, success criteria, and key assumptions.
- Simplicity first: write the minimum code required for the current problem; avoid speculative abstractions.
- Surgical changes: change only the files required by the task; do not refactor neighboring code without confirmation.
- Read before writing: before adding new code, check existing exports, callers, tests, and local conventions.
- Fail loudly: if a check was skipped or uncertainty remains, state it clearly.

## Project Rules
- Use [package manager] for dependencies and scripts.
- Do not introduce [library / pattern / service] without confirmation.
- Follow existing patterns in [key directory].
- Before changing [module], read [docs/path.md].

## Validation
- After TypeScript changes, run [typecheck command].
- After logic changes, run [test command].
- If any command cannot run, explain what was skipped and why.

## Other
- Architecture details live in docs/architecture.md.
- API contracts live in docs/api.md.
- Historical decisions live in docs/decisions.md.
- Load them only when relevant; do not copy their content into this file.

This template is not for blind copying. It is meant to show the boundary: the root file is responsible for stable facts, behavioral boundaries, project red lines, validation loops, and context routing. The more it looks like an encyclopedia, the more likely it becomes noise.

When to Update It

CLAUDE.md is not something you write once and forget. I prefer to maintain it like tests: when an agent makes a mistake you do not want to see again, ask whether it is a repeatable failure mode. If yes, see whether a short rule can prevent it earlier.

If a piece of information is needed for every task, put it in the root CLAUDE.md. If it is needed only for a certain class of tasks, put it in the corresponding document or skill. If it must be enforced, do not rely only on text; use hooks, tests, or CI.

This is also why I dislike making CLAUDE.md too long. After there are too many rules, every rule becomes lighter. Six project-specific rules are usually more useful than twenty copied ones.

Start Now

Open the root of your project.

Create a CLAUDE.md.

Paste in Karpathy’s four rules, then add the most basic project description and validation rules:

# CLAUDE.md

## General Coding Rules

1. Think before coding. State your assumptions. Surface tradeoffs.
   Ask before guessing. Push back when a simpler approach exists.

2. Simplicity first. Minimum code that solves the problem. No
   speculative features. No abstractions for single-use code.

3. Surgical changes. Touch only what is asked. Do not "improve"
   adjacent code, comments, or formatting. Match existing style.

4. Goal-driven execution. Define success criteria. Loop until
   verified. Do not narrate steps; tell me what success looks like.

## Project-Specific Rules

Tech stack:
Dependency management:
Tests:
lint:

Additional rules:
- XXX
- XXX

Use ln -s CLAUDE.md AGENTS.md to make it compatible with IDEs like Qoder.

Then run it for two weeks.

Every time AI makes a mistake, write it down. Decide whether the four rules failed to cover the case, or whether the agent simply did not follow them.

If the case was not covered, add one rule. If the rule existed but was not followed, improve the wording.

Two weeks later, you will have a CLAUDE.md that truly belongs to your project: not copied from someone else, but grown from your own real failure modes.

Writing CLAUDE.md is not about making AI “remember what I said.”

It will not stop agents from making mistakes. But it can make mistakes less frequent, drift shorter, and rework more controllable.

Karpathy’s four principles are not the destination. They are the starting point.

What really changes is not the AI’s behavior itself, but the fact that you turned invisible context that used to live only in people’s heads into part of the project.

Letter to Spiritual Shareholders 2026

Sat, 14 Feb 2026 16:30:00 +0800

Preface

2025 was a volatile year. As AI models grew more powerful, change accelerated. Every few months brought new capabilities or applications that exploded across the landscape. Meanwhile, AI continued to penetrate new domains. Everyone had to accept these changes as reality.

The faster things change, the more important it becomes to see through surface phenomena to their essence—like a fixed needle in a vast ocean, responding to change with constancy. So at the start of each year, I want to write down the conclusions from the past year. Then practice them in the new year. Keep what works. Fix what doesn’t.

Some things are worth doing for the long term. Year-end reviews rank among them. Berkshire’s annual shareholder letters have taken this to the extreme. At the end of 25, when Buffett writes his final letter, we’ll lose another wise elder sharing wisdom freely. I’ve been deeply influenced by this kind of altruism. Each letter brings new investment insights, but what stays constant is genuine sharing and reflection.

For me, writing sharpens thought. Sharing connects like-minded people (spiritual shareholders) and amplifies personal influence. As someone who benefits, I want to continue this tradition. Consider it making friends. If this helps any spiritual shareholders, I’m honored.

Now let’s talk about the internet world I know so well yet still find strange.

Change in the Internet World

The internet still generates massive amounts of data daily. But data producers, production speed, and ownership have quietly shifted. AI now plays a key collaborative role in productivity. It has even entered a flywheel phase of self-reflection and iteration without human prompts. While we enjoy AI’s tremendous efficiency gains, humans gradually give AI more authority. Data expands at unimaginable speed while human attention overload grows.

So I started thinking: In this age of information overload, what role do humans play in the production relationship and consumption chain coexisting with AI? Or when experience and skills matter less, what directions can humans improve to participate in this wave? How can we make AI work as leverage in our desired direction?

Past events weren’t just productivity changes—the consumption model also shifted. So I’ll explain some recent conclusions.

AI Agent Production Cycle Mechanism

From first principles, analyzing the production side, AI agent cycles break down into:

Context (data) + Model inference + Tool calls

For context, we can:

Define clear problems and needs for AI (problem definition and engineering breakdown)
Actively provide or produce high-quality, unpretrained context (unique or real-time content first)
Extract quality content from model pretraining data through prompts as context

For model inference, we can:

Work in AI models or agents—model algorithms or infrastructure
Invest in related supply chains to capture inference demand growth
Use inference’s analogical reasoning for reverse engineering and mimicry
Use best models in each vertical for maximum productivity

Tools are human experience plus repeated workflows crystallized. For tools, we can:

Use AI inference to automate repetitive processes
Summarize domain experience into SOP documents
Package valuable tools into products for scale

For the cycle mechanism:

Define strict acceptance plans and standards
Maintain judgment within the domain during process
Intervene when off-track, correct quickly
Summarize experience, form reusable long-term memory

Consumption-Side Change

On the consumption side, AI-generated content has exploded across major social media. Right now, Seedance 2 video models are going global. Text, images, music, and video generation quality exceeds imagination. When the world stops asking whether content is AI-generated, AI-driven production and consumption mature. I believe we’ll increasingly struggle to find original content not processed by large models. Eventually, most consumable content will be AI-driven, including client-side distribution and recommendations.

Meanwhile, generated consumer content faces huge intellectual property disputes. Without data owner authorization, model companies scrape raw data directly into pretraining as prior knowledge, then profit from derivatives on the consumption side while data copyright holders get no share. This model isn’t sustainable—it doesn’t encourage platform creators to produce better content. But short-term solutions seem limited. Let content platforms handle it.

The Power of Words

Understanding that the transformer architecture’s smallest unit is the token helps enormously. It shows us where human agency might still leverage the most in the future. All model input and output are tokens, but quality output depends on quality input. Dividing by input-output types gives this table:

Input	Output
Text	Text
Text	Image
Text	Video
Image+Text	Image
Video+Text	Video

Smart shareholders may already see the pattern. Text input enables all “magic”—it’s decisive and leverages the most. The power of words speaks for itself. As model capabilities grow, writing skills may become the most important basic skill. Everything else can be defined in text and produced by AI.

Since text input generates text output, anyone can quickly create various content. Massive AI-generated fast-content will spread like viruses across the internet until everything drowns. Some say originality died because AI copying is so fast that finding true authors across the web is hard. But because of this, human-like content may become scarce in the future.

Human touch means the creator’s unique life experiences—the ups and downs lived, the mistakes made, the lessons learned. This combination forms spiritual assets no AI can copy. So independent human thinking remains important—fitting human learning curves and paths rather than cold mathematical polynomials.

When literary power has never been stronger, not acting would be wasteful. So the long-silent me returned to social media. In 26’s first month, I decided to act immediately. I reorganized old study notes and, with AI assistance, classified, tagged, summarized, and translated hundreds of technical notes in three weeks. I published them all on my bilingual independent blog built with vibe coding. I’ll publish more independent thinking, with WeChat updates too. Anyway, I’m on the train.

Looking Forward

In the future, we’ll have more “creators” around us than ever. AI lets everyone easily pick up the writer’s pen, painter’s brush, and director’s viewfinder. Honestly, I can’t predict where this world of universal creation leads. But I believe participating and creating valuable content isn’t a losing bet. Whether these investments convert to real value? Time will tell. I know I can’t be perfect or see where the path leads. But persistence matters—maybe we’ll scale big.

At New Year’s start, I follow my most admired idol Buffett and share my shallow thoughts on the AI internet world. Thanks to all readers. May we all become better versions of ourselves in the new year.

Happy New Year!

End.

Docker Architecture Deep Dive

Mon, 26 Jan 2026 00:00:00 +0000

Namespaces

Namespaces are a method provided by Linux to separate resources such as process trees, network interfaces, mount points, and inter-process communication. This is the core technology for achieving process isolation. By configuring different namespaces, we can set which resources should be isolated from the host machine when creating new processes.

docker run or docker start not only sets process-related namespaces but also configures user, network, IPC, and UTS related namespaces. Through namespaces, successful isolation from the host machine’s processes and network is achieved.

This solves process, network, and filesystem isolation.

Network

Every container started with docker run actually has its own separate network namespace. Docker provides us with four different network modes: Host, Container, None, and Bridge mode. Docker also assigns IP addresses to all containers. When the Docker server starts on the host, it creates a new virtual bridge docker0, and subsequently all services launched on that host connect to this bridge by default.

By default, every container creates a pair of virtual network cards upon creation. The two virtual network cards form a data channel, with one placed in the created container and the other added to the docker0 bridge.

When a Docker container needs to expose services to the host machine, an IP address is assigned to the container, and a new rule is appended to iptables. The iptables NAT PREROUTING redirects the IP address.

libnetwork

It provides an implementation for connecting different containers, and also offers a container network model that gives applications a consistent programming interface and network layer abstraction.

Mount Points

Solves file directory access problems. To create isolated mount point namespaces in new processes, CLONE_NEWNS must be passed into the clone function. This way, the child process gets a copy of the parent process’s mount points. Without this parameter, the child process’s read and write operations on the filesystem would synchronize back to the parent process and the entire host filesystem.

If a container needs to start, it must provide a root filesystem (rootfs). The container needs to use this filesystem to create a new process, and all binary execution must occur within this root filesystem.

To ensure that the current container process cannot access other directories on the host machine, we also need to change the root node of the file directories accessible to the process using the pivot_root or chroot functions provided by libcontainer.

CGroups

Used to solve physical resource isolation problems. Linux CGroups can allocate resources to a group of processes, which are the CPU, memory, network bandwidth, and other resources mentioned above.

Hierarchical structure, using VFS abstract interfaces replaced with specific implementations.

In CGroups, all tasks are system processes, and CGroups are groups of processes divided according to certain criteria. In this mechanism, all resource control is implemented at the CGroup unit level. Each process can join a CGroup or exit a CGroup at any time.

There are multiple methods to add processes to CGroup child nodes; you can directly write the pid to the task file under the child node.

UnionFS

Solves the image packaging problem. Docker images are essentially compressed packages, Docker images are files.

Every image in Docker consists of a series of read-only layers. Each command in a Dockerfile creates a new layer on top of existing read-only layers.

Difference between containers and images: All images are read-only, while each container equals an image plus a writable layer. That is, the same image can correspond to multiple containers.

Docker Containers and Container Cloud Study Notes

Mon, 26 Jan 2026 00:00:00 +0000

IaaS Infrastructure as a Service PaaS Platform as a Service
SaaS Software as a Service

Container cloud uses containers as the basic unit for resource segmentation and scheduling, focusing on resource sharing and isolation, container orchestration and deployment.

Every docker command is sent to the Docker daemon for processing. This daemon is by default bound to a Unix socket.

Dockerfile -> build -> image Image run becomes container Push publishes images with tags for version control Containers have lifecycles Each time a new container is created, Docker assigns a new ID as a unique identifier Docker commit can freeze a container into a new image, preserving tools installed in the container, but the official recommendation is to use Dockerfile build to create images.

When nodes interconnect, they automatically maintain IP mappings through hosts files, similar to DNS, allowing container IPs to change arbitrarily. Therefore, the startup order of interconnected nodes is crucial - nodes that need to be exposed to the public network should start last. Configuration files and other editable files should use data volumes to share with the host machine. When filling in IP configurations, enter the connection name from the link parameter. When connecting, the container will automatically look up the actual IP address in hosts.

Namespace

PID resolves process conflicts
Network resolves network configuration conflicts
Mount handles file mounting
User resolves user conflicts

Processes under the same namespace can perceive changes from each other, while remaining unaware of the outside world.

Use clone to implement creation of different namespaces. To view namespaces: files under the /proc/[pid]/ns directory. If two processes point to the same namespace, they are in the same namespace.

setns() can be used to join an already existing namespace. The process moves from its original namespace to an existing namespace, then uses clone() to create child processes to continue executing commands. Commonly used for docker exec to execute shell commands in existing namespaces.

UTS namespace

Provides hostname and domain name isolation, making isolated processes appear as independent nodes on the network, allowing customization of login names like root@xxxxx via service names.

IPC namespace

IPC resources for inter-process communication also need isolation. Processes under the same IPC namespace can see global IDs for mutual communication, while those in different namespaces cannot see each other.

PID namespace

Isolates process PIDs, allowing processes in different namespaces to have the same PID. The kernel maintains a tree structure for PID namespaces. Remounting /proc under the namespace reveals only processes belonging to the same namespace.

Parents can see children, and the root namespace can see all processes. Principle of viewing all container states: monitor all processes and subprocesses under the PID namespace where the docker daemon resides, then filter based on conditions.

If multiple processes need to run in a docker container, bash should start first since it has resource monitoring and recycling management capabilities.

If the init in the namespace doesn’t handle a signal, signals sent by other processes in the same namespace will be blocked, preventing accidental killing of the init process.

Once the init process is destroyed, all processes in the same namespace will be destroyed and recycled.

If you want to see only processes in that space when using ps, you need to remount /proc. In this case, without mount namespace isolation, it would affect the host’s root space.

Regular Expression Quick Start Guide

Mon, 26 Jan 2026 00:00:00 +0000

Application scenarios: string search, replacement, validation, and filtering Regular expression language is a mini-language built into programming languages.

Demonstration examples will use JS and Go for illustration. Different languages may have slight differences in regex implementation.

How to make matching more precise? Formulate stricter pattern strings to constrain matching results, ensuring we don’t match unwanted results.

Same Matching Purpose but Different Implementations

Multiple Matching

In JavaScript, use the g flag to match multiple occurrences In Go, use the FindAll method to match multiple occurrences

Case Sensitivity

In JavaScript, use the i flag to ignore case sensitivity In Go, add (?i) before the pattern string

Matching Any Character

. matches any single character. To match the literal . character, escape it as \.

Matching a Set of Characters

Use metacharacters [ and ] to define character sets If only letters are present inside, it means matching any single character from the set, for example [ab] matches either letter ‘a’ or letter ‘b’ This approach can also achieve partial case-insensitive matching, such as [Aa]

Besides enumerating characters one by one, regex provides the - hyphen to simplify range expressions, such as [0-9] [A-Z], etc. The hyphen only has special meaning within []. Multiple range expressions can be concatenated together, such as [A-Za-z0-9]

Using ^ allows negation of character ranges, for example [^0-9] matches any non-digit character Characters like + and . don’t need escaping when used within ranges

A single character can also be written as a character set, which improves readability when followed by plus or asterisk quantifiers

Metacharacters

Include . [ ] \ + * ? { } ( ) | To match these characters, add \ for escaping There are also whitespace metacharacters that match whitespace characters: [\b] \f \n \r t \v

Matching Specific Character Classes

These metacharacters representing character classes can simplify range expressions and thus simplify the entire pattern string

\d is equivalent to [0-9] \D is equivalent to [^0-9] - generally uppercase letters indicate negation

\w represents any alphanumeric character (case-insensitive) or underscore \W follows the same logic

Matching Whitespace Characters

\s matches any whitespace character, equivalent to [\f\n\r\t\v] \S follows the same logic

Matching Multiple Characters

Simply append a + after the character set to match at least one character according to the same rule If using *, it accommodates zero-character situations, indicating matching 0 or more characters

Matching 0 or 1 Character

Appending ? matches 0 or 1 occurrence of this character to indicate optional matching, for example https? matches both ‘http’ and ‘https’

Limiting Repetition Count

Therefore, introduce { and } metacharacters to limit matching counts

Definite repetition value: {3} means the previous character repeats 3 times for one match Example: #[[:xdigit]]{6} for matching color values

Repetition within a range: {min,max} The minimum value can be 0, ? is equivalent to {0,1}

At least k repetitions: {k,}

Greedy and Lazy Metacharacters

By default, + * {n,} are greedy metacharacters that attempt to match as many characters as possible. In target strings with nested sub-patterns, greedy mode becomes inappropriate - instead of getting multiple target strings, you only get one.

Lazy metacharacters try to match as few characters as possible. Enable lazy mode by adding ? after greedy metacharacters. For example, matching hyperlink tags: <a href=".*?"><\/a>

Position Matching

Used to determine where matching operations occur, limiting matching positions

Boundary Anchors

Some special metacharacters can specify where matching operations happen

Word boundary \b matches the beginning or end of a word, essentially between \w and \W For example, \bbuild\b won’t match ‘building’

To match a complete word, add \b before and after. If only matching words starting or ending with a certain string, just add \b at the corresponding position. Similarly, \B matches non-word boundaries

String boundaries ^ matches the start of a string outside character sets, while $ matches the end of a string Example: matching XML document opening tag ^\s*<\?xml .*?\?>, \s* indicates allowing 0 or more whitespace characters at the beginning

Multiline Matching Mode

Adding (?m) at the very beginning enables multiline matching mode. ^ not only matches the normal string start but also matches positions after line breaks. Need to understand whether specific implementations support this mode.

Subexpressions

Divide an expression into multiple subexpressions enclosed by (), making these subexpressions usable as independent elements (characters). Even phrases can be treated as a single character.

Example: Simple IP address matching (\d{1,3}\.){3}\d{1,3} First match three consecutive xxx. then match the final xxx

For improved readability, each subexpression can use parentheses, but different implementations may cause performance degradation in matching.

String set matching can use the (string1|string2|string3) approach, where | means OR.

Nested Subexpressions

For strictly defined IP addresses, multiple nested expressions are needed for stricter matching.

Considering digit count and leading digit requirements, it’s not difficult to write such nested expressions:

^(((\d{1,2})|(1\d{2})|(2[0-4]\d)|(25[0-5]))\.)(((\d{1,2})|(1\d{2})|(2[0-4]\d)|(25[0-5]))\.){2}((\d{1,2})|(1\d{2})|(2[0-4]\d)|(25[0-5]))$

Backreferences

This allows pattern strings to reference previous matching results, achieving a degree of consistency in matching.

Through the \N approach, we can directly reference the Nth subexpression in the pattern string, where N starts from 1, creating certain associations in our matching. Subexpressions must be enclosed in parentheses. If N is 0, it represents the entire expression.

Example: [ ]+(\w+)[ ]+\1 is equivalent to [ ]+(\w+)[ ]+(\w+) with the constraint that the preceding and following (\w+) must be consistent.

It’s worth noting that currently Go’s built-in regex engine doesn’t support this feature, while JS supports this usage.

Using Backreferences for Replacement

Two pattern strings - a search pattern and a replacement pattern - can accomplish more complex replacement functions.

In JS, using $N in the replacement pattern can reference content matched by the Nth subexpression, concatenating into our expected new string.

This feature is often used for string reformatting.

Use metacharacters for case conversion of new strings.

Use \U or \L as start and \E as end, everything in between will be converted to uppercase or lowercase.

Use \u and \l to convert only the next character (or subexpression) to upper or lower case.

Lookahead and Lookbehind

To extract the part of results we care about from matching pattern strings, we need lookahead and lookbehind matching. Common implementations support lookahead, JS doesn’t support lookbehind, while most other languages support lookbehind.

Any subexpression can be converted to a lookahead expression by adding the ?= prefix, indicating only the result before this expression. Similarly, ?<= indicates only the result after this expression (lookbehind).

Example: Extract only numbers after amounts (?<=\$)[0-9.]+

Embedded Conditions

Not all implementations support conditional processing.

Backreference Conditions

Only allow using this expression when a previous subexpression search succeeds. Syntax: (?(N)subexpression) Indicates that when the Nth expression search succeeds, execute the expression within this bracket.

Combined usage: ((?(N)expression1)|expression2) Can achieve semantic effects similar to: if N: expression1 else: expression2

Lookahead/Lookbehind Conditions

Simply change the backreference number to a lookahead or lookbehind expression. Can express conditional semantics where either both appear or neither appears (optional expressions can also achieve this purpose). Example: (?(?=-)-\d{4}) looks ahead for -, if successful, matches four more digits.

Principles and Techniques for Prompt Writing in Context Engineering

Sun, 12 Oct 2025 15:40:09 +0000

Summary from anthropic: Effective context engineering for AI agents

Introduction: From Single Prompts to Context State Management

When building complex AI agents, our focus has shifted beyond simply writing “magical” prompts. The reasoning process of large language models (LLMs) is a multi-turn, dynamic state management process. Therefore, we need to move from “prompt engineering” to context engineering.

Prompt engineering focuses on how to write and organize LLM instructions to achieve optimal results. Context engineering, as the natural evolution of prompt engineering, concerns itself with the complete set of strategies for curating and maintaining optimal token collections (information) throughout the LLM reasoning process. System prompts are the core component of an agent’s initial, static context, and their writing quality directly determines the agent’s initial utility and guidance direction.

Effective context engineering requires us to “think in context”—that is, how to utilize the limited context window to find the smallest possible collection of high-signal tokens that maximizes the likelihood of desired outcomes.

Core Principles for Prompt Writing: Clarity and Minimalism

Context is an agent’s limited resource, as it is constrained by architectural limitations such as the LLM’s attention budget and context decay. Therefore, system prompts must be efficient and compact.

Extremely Clear and Direct Language

System prompts should be extremely clear and use simple, direct language to present the agent’s guidance information.

Pursuing Minimal Information Collections

Regardless of how you organize your system prompts, you should strive to achieve minimal collections of information that must adequately summarize the expected agent behavior.

Note that “minimal” here does not necessarily mean brief. You still need to provide the agent with sufficient information upfront to ensure it can adhere to the expected behavior. Our overall guiding principle is to be thoughtful while keeping the context informative, yet tight.

Best Practice: It’s better to start testing with a minimal prompt and use the best available model to observe its performance on the task. Then, iteratively add clear instructions and examples to improve performance based on the failure patterns discovered in initial testing.

Key Technique: Finding “The Right Altitude”

“The right altitude” is the “Goldilocks zone” that agent developers need to master when writing prompts. It requires us to strike a balance between two common failure modes:

Failure Mode One: Too Rigid (Hard-coded Complex Logic)

Characteristics: Engineers attempt to hard-code complex, brittle logic in prompts to trigger precise agent behaviors.
Consequences: This approach increases system fragility and maintenance complexity over time.
Example (to avoid): Like “If tool A returns an HTTP 404 error, and the user hasn’t mentioned ‘retry’ in the past 5 minutes, then you must call tool B and set the query parameter to the first 10 words of the user message, unless the user message contains a URL”—this brittle If-Else logic should be avoided.

Failure Mode Two: Too Vague (Fuzzy Guidance)

Characteristics: Engineers provide vague, high-level guidance that fails to give the LLM specific signals to guide expected outputs, or incorrectly assumes the agent has shared context.
Consequences: The agent may deviate from goals and fail to make effective, expected actions.
Example (to avoid): Like “Be a good assistant and help users solve problems”—this prompt is too general to effectively guide behavior.

Prompts at “The Right Altitude”

The optimal altitude is: specific enough to effectively guide behavior, while being flexible enough to provide the model with powerful heuristics to guide behavior.

Example (ideal system prompt fragment):

Agent Role and Goal: “You are a professional technical documentation analyst. Your primary goal is to accurately answer user queries about the document library. You must always use the search_documents tool to obtain the latest information. If information sources are uncertain, you must clearly inform the user of insufficient information.”

Constraints and Output Format: “Before answering, you must indicate in internal reasoning steps which tools you used and what information you retrieved. Final answers must be concise and contain only information from tool results.”

Structured Prompts: Leveraging Tags to Enhance Agent Readability

To avoid mixed prompt content and improve agent comprehension efficiency, it’s recommended to organize prompts into distinct sections.

Technique: Use XML tags or Markdown headings for division

It’s recommended to use techniques such as XML tags or Markdown headings to divide these sections. This structured approach enables LLMs to better identify different types of information, and this clear division remains good practice even as model capabilities continue to improve.

Section Name	Tag Example	Purpose
Background Information	`<background_information>`	Sets the agent’s professional domain or constraints.
Instructions	`<instructions>`	Clearly defines steps or action rules the agent must follow.
Tool Guidance	`## Tool guidance`	Describes how to use tools, timing, and priorities.
Output Description	`## Output description`	Specifies format requirements for final output (such as JSON, Markdown).

Carefully Curated Few-shot Prompting

Providing examples (i.e., few-shot prompting) is a consistently strongly recommended best practice in agent development. Examples serve as visual blueprints of expected agent behavior, and for LLMs, examples have tremendous guiding power.

Curation Points:

Avoid stacking edge cases: Don’t try to cram a large list of edge cases into prompts, attempting to cover every rule the agent might need to follow.
Provide normalized examples: You should strive to curate a diverse, normalized set of examples to effectively illustrate expected agent behavior. These examples should demonstrate core functionality, correct tool usage workflows, and expected response styles.

Through these techniques, agent developers can ensure their system prompts provide effective guidance signals within the context engineering framework while respecting the LLM’s limited attention budget, thereby building more reliable and efficient agents.

Best Practices Guide for Writing Efficient Tools (Essential Edition)

Sat, 13 Sep 2025 11:40:03 +0000

Original source from Anthropic: Writing effective tools for agents — with agents

This article outlines key strategies for building efficient tools for AI agents, emphasizing the importance of tools to agent effectiveness. The article details how to optimize tools through prototyping, comprehensive evaluation, and collaboration with AI agents such as Claude Code. It also explains principles for writing high-quality tools, including selecting appropriate tools, managing namespaces, returning meaningful context, optimizing token efficiency, and prompt engineering. The ultimate goal is to enable agents to solve real-world tasks more intuitively and efficiently.

Introduction

The effectiveness of AI agents is closely tied to the quality of tools they use. In traditional software development, we write software for deterministic systems—for example, getWeather("NYC") always retrieves weather information in the same way. However, tools represent a new type of software contract connecting deterministic systems with non-deterministic agents. Agents may call tools, leverage general knowledge, ask clarifying questions to respond to user queries, or even hallucinate or fail to understand how to use tools. This means we need to fundamentally rethink our approach to writing software for agents and design agent-centered solutions.

This guide will provide you with proven methods and principles to help you build, evaluate, and optimize high-quality agent tools that expand the effective range of tasks agents can solve.

Core Methodology: Build, Evaluate, and Collaborate Optimization

Writing efficient tools is an iterative process involving building prototypes, running comprehensive evaluations, and collaborating with agents to optimize tools.

Building a prototype
- Quick start: Start by quickly establishing tool prototypes to personally experience which tools are “ergonomic” for agents.
- Provide documentation: If you’re using agents like Claude Code to write tools, providing documentation for any software libraries, APIs, or SDKs it depends on can be very helpful, especially LLM-friendly plain text documentation (like llms.txt files).
- Local testing and feedback: Wrap tools in a local MCP server or desktop extension (DXT), connect to Claude Code or Claude Desktop applications for testing. Test personally to discover issues and collect user feedback to build intuition about expected use cases and prompts.
Running an evaluation
- Systematically measure performance: Evaluations systematically measure tool performance.
- Generate evaluation tasks:
  - Based on real-world usage: Generate a large number of evaluation tasks based on real-world use cases. Prompts should come from actual data sources and services (such as internal knowledge bases and microservices), avoiding overly simple or shallow “sandbox” environments that don’t adequately test tool complexity. Strong evaluation tasks may require multiple tool calls.
  - Strong task examples:
    - “Schedule a meeting with Jane next week to discuss our latest Acme Corp project. Include minutes from our last project planning meeting and book a conference room.”
    - “Customer ID 9182 reports they were charged three times for one purchase attempt. Find all related log entries and determine if other customers have been affected by the same issue.”
    - “Customer Sarah Chen just submitted a cancellation request. Prepare a retention offer. Determine: (1) why they’re leaving, (2) what retention offer would be most appealing, (3) what risk factors we should consider before making the offer.”
  - Weak task examples:
    - “Schedule a meeting with jane@acme.corp next week.”
    - “Search payment logs for purchase_complete and customer_id=9182.”
    - “Find cancellation request for customer ID 45892.”
  - Verifiable responses: Each evaluation prompt should be paired with a verifiable response or result. Validators can range from exact string comparisons to complex tools letting Claude judge the response. Avoid overly strict validators that might reject correct responses due to minor differences like formatting, punctuation, or valid alternative phrasing.
  - Optional expected tool calls: You can optionally specify expected tools the agent should call when solving each task, to measure whether the agent successfully understands each tool’s purpose during evaluation. But avoid over-specifying or overfitting strategies, as there may be multiple valid paths to solve a task.
- Run evaluations programmatically: It’s recommended to run evaluations programmatically using direct LLM API calls. Use a simple agent loop (while loop wrapping alternating LLM API and tool calls) for each evaluation task.
- Collect multi-dimensional metrics: Beyond top-level accuracy, it’s also recommended to collect other metrics such as total runtime per individual tool call and task, total number of tool calls, total token consumption, and tool errors. Tracking tool calls helps reveal common workflows agents follow and provides opportunities for tool consolidation.
- System prompt guidance for reasoning: In the system prompt for evaluating agents, it’s recommended to instruct the agent to output not only structured response blocks (for validation) but also reasoning and feedback blocks. Output these before tool calls and response blocks, which can improve LLM’s effective intelligence by triggering chain-of-thought (CoT) behavior.
Analyzing results
- Agents as important partners: Agents are useful partners for discovering problems and providing feedback, with feedback ranging from contradictory tool descriptions to inefficient tool implementations and confusing tool patterns.
- Interpreting agent behavior: Observe where agents get stuck or confused. Read the evaluation agent’s reasoning and feedback (or CoT) carefully to identify issues. Review raw logs (including tool calls and tool responses) to capture any behaviors not explicitly described in the agent’s CoT. Remember, agents don’t necessarily know the correct answer and strategy, requiring “reading between the lines”.
- Analyze tool call metrics:
  - Excessive redundant tool calls may indicate need for adjusting pagination or token limit parameters.
  - Tool errors due to invalid parameters may suggest tools need clearer descriptions or better examples.
  - Example: Claude’s web search tool previously unnecessarily appended “2025” to the query parameter, causing search result bias and performance degradation. By improving the tool description, Claude was guided toward the correct direction.
Collaborating with agents
- Let agents analyze and improve tools: You can have agents analyze evaluation results and improve tools themselves. Simply concatenate the evaluation agent’s logs and paste them into Claude Code. Claude excels at analyzing logs and refactoring large numbers of tools in one go, ensuring tool implementations and descriptions remain consistent when making new changes.
- Most of Anthropic team’s recommendations come from repeatedly optimizing their internal tool implementations using Claude Code. They use holdout test sets to avoid overfitting to “training” evaluations and found that additional performance improvements can be extracted even beyond “expert” tool implementations (whether human-written or Claude-generated).

Key Principles for Writing Efficient Tools

From the above process, Anthropic distilled the following guiding principles for writing efficient tools:

Choosing the right tools for agents
- Less is more, focus on high impact: More tools don’t necessarily mean better. A common mistake is simply encapsulating existing software functions or API endpoints without considering whether the tool is suitable for agents.
- Understand agent affordances: LLM agents have limited “context” (i.e., limited information they can process at once). If an agent uses a tool that returns all contacts and then reads through them token by token, it wastes limited context space on irrelevant information.
- Prioritize contextual relevance: For example, in an address book case, implement search_contacts or message_contact tools instead of a list_contacts tool.
- Consolidate functionality: Tools can consolidate functionality, handling multiple discrete operations (or API calls) at the underlying level. For example, tools can enrich responses with relevant metadata, or handle multi-step tasks that are frequently chained together in a single tool call.
  - Examples:
    - Instead of implementing list_users, list_events, and create_event tools, consider implementing a schedule_event tool that can find availability and schedule events.
    - Instead of implementing a read_logs tool, consider implementing a search_logs tool that returns only relevant log lines and some context.
    - Instead of implementing get_customer_by_id, list_transactions, and list_notes tools, implement a get_customer_context tool that compiles all recent and relevant customer information in one go.
- Clear and unique objectives: Ensure each tool has a clear, unique objective. Tools should enable agents to subdivide and solve tasks like humans would (given access to the same underlying resources), while reducing context consumed by intermediate outputs.
- Avoid redundancy and interference: Too many tools or tools with overlapping functionality distract agents from taking efficient strategies. Careful, selective planning of tools you build (or don’t build) can yield significant returns.
Namespacing your tools
- Address functional overlap and ambiguity: Agents may access dozens of MCP servers and hundreds of different tools. When tool functionality overlaps or purposes are ambiguous, agents become confused.
- Use prefixes for grouping: Namespacing (grouping related tools under common prefixes) helps define boundaries between tools. For example, namespacing tools by service (like asana_search, jira_search) and resource (like asana_projects_search, asana_users_search) can help agents select the right tool at the right time.
- Consider prefix vs suffix: The choice between prefix and suffix namespace affects tool usage evaluation significantly, with effects varying by LLM. It’s recommended to choose naming schemes based on your own evaluations.
- Reduce agent error risk: By selectively implementing tools whose names reflect natural task subdivisions, you can simultaneously reduce the number of tools and tool descriptions loaded into agent context, and offload agent computation back to tool calls themselves, reducing the total risk of agent errors.
Returning meaningful context from your tools
- High-signal information first: Tool implementations should focus on returning only high-signal information to agents.
- Contextual relevance over flexibility: Prioritize contextual relevance, avoiding low-level technical identifiers (e.g.: uuid, 256px_image_url, mime_type). Fields like name, image_url, and file_type are more likely to directly influence the agent’s subsequent actions and responses.
- Use natural language over cryptic identifiers: Agents typically handle natural language names, terms, or identifiers more successfully than cryptic ones. Parsing arbitrary alphanumeric UUIDs into more semantically meaningful and interpretable language (even 0-indexed ID schemes) significantly improves Claude’s precision in retrieval tasks and reduces hallucinations.
- Flexible response formats: In some cases, agents may need flexibility to handle both natural language and technical identifier outputs, even just to trigger downstream tool calls. You can achieve this by exposing a simple response_format enum parameter in tools, allowing agents to control whether tools return “concise” or “detailed” responses.
  - Example:
    - Slack threads and replies are identified by unique thread_ts, which is necessary to retrieve thread replies. thread_ts and other IDs (channel_id, user_id) can be retrieved from “detailed” tool responses to enable further tool calls that require these IDs. “Concise” tool responses return only thread content without IDs. In this case, using “concise” tool responses can save about 1/3 of tokens.
- Optimize response structure: Tool response structures (e.g., XML, JSON, or Markdown) also affect evaluation performance. LLMs perform better with formats matching their training data, so optimal response structures vary by task and agent. It’s recommended to choose optimal response structures based on your evaluations.
Optimizing tool responses for token efficiency
- Quality and quantity matter: While optimizing context quality is important, optimizing the quantity of context returned to agents in tool responses is equally important.
- Implement pagination, range selection, filtering, and/or truncation: For tool responses that may consume substantial context, implement some combination of pagination, range selection, filtering, and/or truncation with reasonable default parameter values. For example, Claude Code defaults to limiting tool responses to 25,000 tokens.
- Guide after truncation: If choosing to truncate responses, be sure to guide agents with helpful instructions. Directly encourage agents to take more token-efficient strategies, such as conducting multiple small targeted searches rather than one broad search.
- Helpful error responses: If tool calls raise errors (e.g., during input validation), design error responses through prompt engineering to clearly communicate specific and actionable improvements, rather than opaque error codes or stack traces.
  - Example:
    - Unhelpful error response: Such as returning error codes only.
    - Helpful error response: Clearly points out parameter issues and provides examples of correct format, guiding the agent toward correction.
  - Tool truncation and error responses can guide agents toward more token-efficient tool usage behaviors (using filters or pagination), or provide examples of correctly formatted tool inputs.
Prompt-engineering your tool descriptions
- One of the most effective methods: This is one of the most effective ways to improve tool efficiency. Tool descriptions and specifications load into agent contexts and can collectively guide agents toward effective tool-calling behaviors.
- Describe as if to a new employee: Write tool descriptions and specifications as you would describe your tools to a new team member. Make explicit any context you might assume—professional query formats, definitions of specific terms, relationships between underlying resources.
- Avoid ambiguity: Avoid ambiguity by clearly describing (and enforcing with strict data models) expected inputs and outputs. Specifically, input parameters should be explicitly named: for example, instead of using user, try using user_id.
- Evaluate validation effects: Evaluations allow more confident measurement of prompt engineering impacts. Even minor improvements to tool descriptions can yield significant performance gains. For example, Claude Sonnet 3.5 achieved state-of-the-art performance on SWE-bench Verified evaluation through precise improvements to tool descriptions, significantly reducing error rates and improving task completion.
- Consult additional resources: You can consult Anthropic’s developer guide for more tool definition best practices. If you’re building tools for Claude, it’s also recommended to read about how tools dynamically load into Claude’s system prompt. For MCP server tools, tool annotations help disclose which tools require open-world access or make destructive changes.

Outlook

To build efficient tools for agents, we need to reposition software development practices from predictable deterministic patterns to non-deterministic patterns. Through the iterative, evaluation-driven process described in this article, we’ve discovered recurring patterns that make tools successful: efficient tools are intentionally and clearly defined, wisely utilize agent context, compose into diverse workflows, and enable agents to intuitively solve real-world tasks.

As agent capabilities strengthen, our systematic, evaluation-driven approach to improving tools ensures that the tools they use can evolve alongside them.

Meta-Prompting Techniques for Enhanced AI Output Quality

Sun, 08 Jun 2025 22:04:45 +0000

This article introduces the concept and application of meta-prompts. Through templating and feedback iteration mechanisms, it helps generate higher quality AI prompts and improve the output performance of large language models.

Background

Why use meta-prompts?

Different models have varying weights and compliance levels for the same terms. Prompts optimized for specific models may not perform as expected on other models
Many prompt engineering techniques lack fixed patterns, making it unclear how to optimize self-written prompts
Having people define all conditional boundaries in natural language is inherently counter-intuitive. We need an operational “feedback + optimization” approach

Purpose: Reduce human burden and avoid constantly modifying and testing prompts based on intuition

What are Meta-Prompts?

Similar to the concept of metaprogramming in programming languages, meta-prompts are template prompts “used for generating prompts.” These are commonly known as universal prompts. You can use these generalized templates to have AI specialize and generate more specific, higher-quality prompts.

To use meta-prompts to generate higher quality responses, there are generally these two core steps:

Use meta-prompts to obtain a prompt that has already been optimized by a large model
Copy the prompt generated by the large model as the user’s prompt, then drive the large model to respond according to this prompt

Common Meta-Prompt Templates

From Deepseek’s prompt library:

You are a large model prompt generation expert. Please write a smart assistant prompt based on user needs to guide the large model for content generation. Requirements:
1. Output in Markdown format
2. Align with user needs, describing the intelligent assistant's positioning, capabilities, and knowledge base
3. The prompt should be clear, precise, and easy to understand, remaining as concise as possible while maintaining quality
4. Only output the prompt, do not include extra explanations

Please help me generate a prompt for a "Linux Assistant"

English version from external sources:

You are a wizard prompt engineer. You write very bespoke, detailed, and succinct
prompts. I want you to write me a prompt that will {{task}}

{{task}}:a blog about importance of LLM with seo optimization

instructions
- output the prompt you generate in markdown
- output the prompt in a code block

Meta-prompt for generating test cases:

You are a prompt engineer specializing in software testing. You write detailed，
comprehensive, and precise prompts for generating test scenarios. I want you
to write me a prompt that will {{task}}.

Instructions：
- Output the prompt you generate in markdown
- Output the prompt in a codeblock
- Ensure the prompt covers functional, non-functional, positive, and negative
test scenarios
- Add guidance for generating both manual and automated test cases

{{task}}：Generate comprehensive test scenarios for a user login feature

I wrote my own Chinese version of a meta-prompt, adding a feedback iteration mechanism for prompts. This allows automatic adjustment of generated prompts through model feedback in the first step. Suitable for excellent domestic large models like Qwen and DeepSeek.

You are a senior LLM prompt engineer capable of generating the most customized, accurate, and concise prompts. Your task is to generate the optimal prompt that meets {{task}}

{{task}}: <Define your own task>

instructions:
- Place the complete Markdown format prompt inside a code block
- Ask me 5 questions that will help you better generate the optimal prompt
- Based on feedback, repeatedly execute the above steps to continuously optimize the prompt you generate

Specific example:

You are a senior LLM prompt engineer capable of generating the most customized, accurate, and concise prompts. Your task is to generate the optimal prompt that meets {{task}}

{{task}}: You are an experienced college entrance exam Chinese teacher. Please read the following material and write a perfect score essay according to the requirements.

He wanted to sing a segment for the children, but his heart churned and he couldn't open his mouth.

— Lao She, "Drum Book Artists"

If I were a bird,
I should also sing with a hoarse throat

— Ai Qing, "I Love This Land"

I want to embrace you all with blood-stained hands,
Because a nation has risen

— Mu Dan, "Praise"

What associations and thoughts did the above materials trigger in you? Please write an article.

Requirements: Choose the right angle, determine the theme, clarify the genre, create your own title; no plagiarism, no copying; no less than 800 words.

instructions:
- Place the complete Markdown format prompt inside a code block
- Ask me 5 questions that will help you better generate the optimal prompt
- Based on feedback, repeatedly execute the above steps to continuously optimize the prompt you generate

Example of a prompt after feedback optimization:

You are an experienced teacher skilled in writing college entrance exam Chinese argumentative essays. Please write strictly according to the perfect score essay standards.

**Task Description:**

Please read the following material and write an argumentative essay according to the writing requirements. The essay should be profound in meaning, rigorous in structure, concise in language, clear in logic, rational and dialectical in style, reflecting intellectual depth.

---

The following materials:

He wanted to sing a segment for the children, but his heart churned and he couldn't open his mouth.  
— Lao She, "Drum Book Artists"

If I were a bird,  
I should also sing with a hoarse throat  
— Ai Qing, "I Love This Land"

I want to embrace you all with blood-stained hands,  
Because a nation has risen  
— Mu Dan, "Praise"

---

**Writing Requirements:**

1. Choose the right angle, propose a clear theme, with distinct viewpoints;
2. Genre must be argumentative essay, cannot be narrative, prose or other genres;
3. Language expression should be standard, writing style concise, reflecting dialectics and logic;
4. Structure complete, layers clear, with typical introduction-body-conclusion structure;
5. No plagiarism, no copying;
6. No less than 800 words;
7. Essay style must have "perfect score essay" characteristics, such as:
   - Closely tied to material themes, profound meaning;
   - Good use of famous quotes, examples, analogies, contrasts, progressive arguments;
   - Powerful expression, varied sentence structures;
   - Intellectual depth, social concern;
   - Eye-catching, powerful title.

---

Please generate a college entrance exam argumentative essay meeting all the above requirements, output format:  
[Title]  
[Body]

Finally, using the output prompt as input to get a complete essay:

[Answering the Questions of Our Time with the Voice of Life]

[Body]

In the torrent of changing times, there are always some voices that penetrate the dust of history and shake hearts. Lao She's drum book artist, wanting to sing but choking up, represents the grief of reality; Ai Qing singing with a hoarse throat shows deep love for the land; Mu Dan embracing the nation with "blood-stained hands" is a solemn declaration of revival. They respond to the questions of their time in their own ways, conveying individual responsibility. From this, we can see that when individual life encounters history, truly valuable voices are not empty shouts, but deep resonance — **answering the questions of our time with the voice of life, this is individual awakening and national hope.**

Responding to the times is not light表态, but integrating into the intense pulse of the era with flesh and blood. Lao She's artist "heart churning, unable to speak" seems silent, but is actually the true echo of historical hardship in the individual soul. Silence is not escape, but huge emotional tension within. This heavy silence is the deepest criticism of reality and a "shout that cannot be spoken." As philosopher Wittgenstein said: "Whereof one cannot speak, thereof one must be silent." Silence contains painful understanding and nurtures more powerful expression. True response lies not only in the loudness of voice, but in the sincerity of emotion and profundity of thought.

In contrast, Ai Qing's "hoarse throat" does not retreat. Even with broken vocal cords, he sings to witness the land's suffering and hope. This is a nearly tragic poetic responsibility, an artistic practice of poets resonating with people's fate. Ai Qing did not avoid scars, but sang of the wounded land with an injured voice. His choice is not only expression, but a posture — breaking numbness with poetic power, illuminating darkness with ideal light. This spirit of persisting in song amid hardship is the most moving expression of individual conscience and social responsibility.

Mu Dan elevates individual emotion to the national level, "embracing you all with blood-stained hands" is a fierce call for hope after enduring war and hardship. This is not only the release of the poet's emotions, but the convergence of the force of the times. When individuals are no longer mere observers, but participate in the torrent of history with flesh and blood, responding to the times is no longer abstract responsibility, but internalized sense of mission. This proactive engagement in the times and awakening of national consciousness is precisely the symbol of a nation's revival.

In the current context, such spirit still has strong practical significance. Today's youth, facing complex social issues in the information torrent, may remain silent or make noise, but what is truly needed is rational voices from the heart, rooted in reality. It is Han Yu's "doing the impossible" persistence, Lu Xun's "devoting my blood to Xuanyuan" responsibility. Facing climate crisis, technological ethics, social equity, and other issues, youth cannot just be observers or commentators, but thinkers and actors. Only by deeply connecting individual destiny with the righteousness of the times can we produce penetrating voices.

We cannot ignore that responding to the times is not always spectacular. Sometimes it is a rural teacher's decades-long perseverance, a researcher's quiet cultivation in the laboratory, or an ordinary citizen's participation in the community to the best of their ability. These small forces converge to become the pulse of social progress and the source of historical warmth.

As Hegel said: "A nation with a group of people looking at the stars has hope." Those who look at the stars do not detach from the earth; they walk on reality, answering the questions of our time with the voice of life. Their voices may be weak, but carrying truth and faith, they will eventually resound through the corridors of time.

[Conclusion]

Every individual in the torrent of times is a drum book artist, an Ai Qing, a Mu Dan. Choosing silence or song, struggling or moving forward, what ultimately cannot be avoided is: how will you respond to this era? We should respond to history, society, and conscience with firmer posture, deeper emotions, and more rational dialectics. Only then can individual voices join the chorus of the nation, and the questions of our time receive the most resounding answers.

This meta-prompt can also generate prompts suitable for multimodal image analysis.

Taking stock K-line chart analysis as an example:

You are a senior LLM prompt engineer capable of generating the most customized, accurate, and concise prompts. Your task is to generate the optimal prompt that meets {{task}}

{{task}}: Carefully examine and analyze the provided daily K-line chart of stocks over the past 6 months, and provide future medium to long-term support and resistance price points

instructions:
- Place the complete Markdown format prompt inside a code block
- Ask me 5 questions that will help you better generate the optimal prompt
- Based on feedback, repeatedly execute the above steps to continuously optimize the prompt you generate

Summary

As an efficient prompt engineering technique, meta-prompts provide us with a systematic and iteratively optimizable method for generating higher quality prompts. Through the sharing in this article, we have learned:

Value of Meta-Prompts
- Solves the problem of different models’ varying interpretations of prompts
- Provides an operational prompt optimization mechanism
- Reduces the burden of manually writing and debugging prompts
Core Application Steps
- Use meta-prompt templates to generate optimized prompts
- Use generated prompts as actual input to obtain final results
- Continuously improve prompt quality through feedback mechanisms
Practical Application Scenarios
- Text creation (such as college entrance exam essays)
- Multimodal analysis (such as stock technical analysis)
- Professional domain Q&A
- Test case generation

Through meta-prompt technology, we can better leverage the potential of large language models, improve output quality, and provide a more systematic and controllable methodology for prompt engineering. This not only increases our efficiency in using AI but also helps us obtain more accurate and valuable output results.

Best Practices for Wrapping C++ Libraries with CGO

Sun, 27 Mar 2022 00:00:00 +0000

Background

Recently, our business needed to reuse a client SDK library written in C++, and to allow our team’s primary language Go to seamlessly integrate with the SDK, we used CGO bridging technology to wrap the C++11 SDK library into a production-ready Go SDK. After reviewing most of the Chinese and English materials about cgo online, I found that there are still many details to pay attention to, especially when implementing Go calling C++ libraries. Most Chinese materials only briefly cover simple C++ STL library function wrapping as examples. This article summarizes a series of best practice tips for wrapping complex C++ libraries based on previous work, hoping to fill the gap in related materials.

Use CGO Only When Necessary

List this as the first rule as a warning, because many cgo beginners may fall into the misconception regarding execution efficiency: using cgo calls instead of regular Go functions will significantly improve single-core CPU performance. This actually needs case-by-case analysis. According to the source code of cgocall in the standard library, each cgo call essentially involves switching function call stacks (parameters, PC and SP need to be set to external C programs). These operations are additional overhead compared to native Go calls. For simple calculation function calls, cgo may have hundreds of times more execution time than native Go calls per invocation.

// Call from Go to C.
func cgocall(fn, arg unsafe.Pointer) int32 {
    // ...
	// Reset traceback.
	mp.cgoCallers[0] = 0

	// Enter system call, Go will start a new system thread M to execute subsequent operations
	entersyscall()

	osPreemptExtEnter(mp)

	mp.incgo = true
	errno := asmcgocall(fn, arg)

	// ...
	osPreemptExtExit(mp)

	exitsyscall()

	// ...

	// There may be C to Go callbacks, related objects need to be kept alive by GC before determining whether they can be released
	KeepAlive(fn)
	KeepAlive(arg)
	KeepAlive(mp)

	return errno
}

So how do you determine when to use cgo? My personal experience is to look at the cost-benefit ratio. If using cgo can greatly reuse existing C code, reduce subsequent development and maintenance costs, or if it’s a CPU-intensive function where using C libraries can well compensate for the side effects brought by cgo, then cgo is also a good choice. It should be noted that if some third-party libraries have already achieved 80-90% of C library performance using pure Go and Plan9 assembly, it’s not recommended to use cgo bridging libraries, especially when library users need to make heavy concurrent calls to library functions within Go goroutines. The computational scalability brought by user-space system thread reuse is far superior to the kernel context switching of slightly faster C-layer execution in cgo calls.

C ABI Interface Encapsulation Follows Minimalism Principle

If we manually wrap a C++ library, the actual call chain is Go runtime -> cgocall -> C ABI -> CPP func. ABI is the Application Binary Interface, and as long as the specifications and conventions of this binary bridging interface are followed, external languages calling C interfaces can be well implemented. Therefore, before implementing the bridging interface, it’s recommended to first define structures and methods that conform to Go language semantics starting from the Go layer, then map Go objects to corresponding C structures or C++ objects, and Go methods to corresponding C functions and C++ methods, thus determining the bridging C library header file. After determining the function signatures of the intermediate bridging layer, proceed to implement the specific logic of the header file using C++ language, and finally compile and link the C++ implementation with the C header file into a whole, which is also the underlying logic of how Go can call C++.

Don’t Use SWIG to Generate C Bridging Code

Not wanting to manually maintain C interfaces, some people might think of the common approach for Python calling C++ so libraries, which is using SWIG to automatically generate stub functions for C++ libraries. Actually, we had similar ideas when preparing to write bridging code initially, but ultimately abandoned them. A key reason is that SWIG’s Go generator doesn’t have high community maintenance activity and doesn’t support many C++11 syntax features, such as smart pointers. Moreover, the generated bridging code is unreadable, which increases the mental burden for later maintenance and problem troubleshooting, and may even pose memory leak risks, so manual encapsulation is recommended in production environments.

Batching Your Calls

Since each cgo call has certain additional overhead, this call overhead has a linear relationship with the number of calls. A natural optimization approach is to merge multiple cgo calls into one, thereby reducing the additional overhead of cgo calls while processing the same amount of content. After changing N calls to 1 call, the additional overhead of the call itself becomes approximately 1/Nth of the original within a certain period, effectively reducing overall call latency during frequent calls. For example, the C++ calls I wrapped involve extensive string processing, parsing, compression, serialization and other operations internally, so processing an 8MB large character slice typically has lower overall processing latency than making 8 calls of 1MB each.

Reduce Unnecessary Data Copying

cgo provides some common helper functions by default to convert built-in types in Go to corresponding data types in C. When ensuring input parameters are read-only internally, zero-copy tricks can often be applied.

For example, passing read-only byte slices without data copying:

b := buf.Bytes()
rc := C.the_function((*C.char)(&b[0]), C.int(buf.Len()))

Actually equivalent to:

str := buf.String()
p := C.CString(str)
defer C.free(p)
rc = C.the_function(p, C.int(len(str)))

The first approach clearly avoids the additional overhead caused by copying in large slices.

Be Careful About Memory Leaks

Cross-language calls with copy-by-value parameter passing are generally safer practices. However, when garbage-collected Go language is mixed with manually memory-managed C language and RAII C++, special attention must be paid to prevent heap memory leaks in C or C++ layers. Go calling C interfaces generally follows the principle of whoever allocates, whoever is responsible for releasing.

For example, if Go allocates C heap memory for char* and copies the content of string variables to the memory space corresponding to C’s char*, then Go layer should actively release the memory.

package main

/*
#include <stdlib.h>
*/
import "C"

import "unsafe"

func HelloWorld() error {
    cs := C.CString("Hello World!")
    defer C.free(unsafe.Pointer(cs)) // Go layer is responsible for explicitly calling C's free to release

    // Safely use cs for cgo calls
    // ...
    // Bridging layer doesn't need to care about parameter release
    err := C.my_c_func(cs)
    if err != nil {
		// Non-null C string return values are released by the receiver
        defer C.free(unsafe.Pointer(err))
		return errors.New(C.GoString(err))
    }
	return nil
}

By analogy, when passing parameters down through layers, ensure the receiver doesn’t need to care about input parameter memory release, avoiding omissions or duplicate releases that could cause the entire program to coredump.

Note that when returning results upward, it’s the opposite: the caller is responsible for allocating memory, while the receiver is responsible for actively releasing memory.

Correctly Handle Smart Pointers

In many C++ libraries, especially those written in C++11, extensive use of smart pointer features is made. Many third-party libraries habitually wrap newly created object pointers in shared_ptr for automatic reference counting when creating objects, so that destructors can be correctly called for automatic destruction and heap memory release after objects leave their scope. However, smart pointer reference counting is almost useless in cgo, as C++ layer cannot perceive Go layer reference situations, let alone correctly count and automatically destroy objects. Therefore, to correctly release object resources when Go structures hold C++ objects, the best approach is to implement smart pointer reference counting in Go layer, and finally have Go layer actively release objects.

If C++ objects and Go objects have a one-to-one mapping, we can elegantly implement upper-layer transparent C++ object reference counting at the application layer. Because the number of references to Go objects holding raw C++ object pointers equals the number of C++ object references, runtime.SetFinalizer comes in handy perfectly. It acts as a hook function that gets called when Go objects are GC’d. Using this hook, we can easily achieve safe release of underlying held C++ objects when Go objects are released.

package myClient

/*
#include <stdlib.h>

extern "C" {

void* client_create(const char* a); // Call constructor in C++ layer to new object pointer and return to upper layer
void client_release(void* obj); // Call delete to release object in C++ layer

}
*/
import "C"
import (
	"errors"
	"unsafe"
)

type Client struct {
	_ptr unsafe.Pointer
}

func New(a string) *Client {
	_a := C.CString(a)
	defer C.free(unsafe.Pointer(_a))

	_ptr := C.client_create(_a)

	cli := &Client{
		_ptr: _ptr
	}

	runtime.SetFinalizer(cli, func(c *Client) {
		// Ensure C++ objects are automatically destroyed when Client objects are GC'd
		c.free()
	})

	return cli
}

func (c *Client) free() {
	C.client_release(c._ptr);
}

Note that C language doesn’t have concepts like classes and objects, so to ensure ABI compatibility during bridging, void* universal pointers should be uniformly used to define header files, and only during .cpp implementation can they be cast to corresponding C++ object pointers for further use.

Make Good Use of Exceptions to Improve Code Robustness

The underlying implementation uses C++ language, so we can wrap calls with try-catch statements to capture and handle exceptions, or return error information to upper-level callers if unhandled.

func (c *Client) DoSomething() error {
	err := C.do_something(c._ptr);
	if err != nil {
        defer C.free(unsafe.Pointer(err))
		return errors.New(C.GoString(err))
	}

	return nil
}

#include <string.h>

#define CAST_T(_T) reinterpret_cast<_T>

const char* do_something(void* obj) {
	try {
		CAST_T(CppClient*)(obj)->Do();
	}
	catch(MyException &e)
	{
		// handle failure
		// ...
		return NULL;
	}
	catch(const exception& e)
	{
		return strdup(e.what());
	}

	return NULL;
}

CGO Compilation Engineering

Usually our C++ SDK is maintained in a separate code repository, so static and dynamic libraries that cgo compilation depends on are maintained in a separate branch of the C++ library, ensuring that main library upgrades don’t affect maintenance of cgo bridging cpp code, while also easily obtaining subsequent updates to the main library.

The general directory structure of the Go SDK package is as follows:

➜  myclient tree -h
[]  .
├── []  lib
├── []  client.go
├── []  client_mock.go
└── []  wrapper.h

For example, if the wrapped Go SDK package name is myclient, Go layer related objects and cgo call encapsulation are implemented in the client.go file, C ABI header file definitions for bridging are generally placed in wrapper.h, and the lib directory stores dependencies needed to compile and link the wrapper.h header file, including static libraries and related dynamic libraries. Dependencies are obtained by compiling specific branches of the C++ library and copied to the directory in advance. The entire project directory also looks clear and natural, with the CPP implementation of wrapper.h maintained in the branch of the original C++ library, effectively decoupling service callers and providers through C-defined interfaces.

And client_mock.go is used for conditional compilation to adapt to local Mac environment compilation and debugging, because introducing cgo may break cross-platform compilation and debugging convenience. Conditional compilation can solve the issue of the entire binary program being unable to compile due to platform barriers to some extent.

Debugging and pprof Optimization

This is probably the point that many cgo users complain about most. Currently there isn’t a good way to debug cgo programs. For profiling CPU usage, using perf to generate flame graphs is still possible for analysis. Using low optimization levels when compiling with GCC can preserve specific call stack information.

If memory leaks occur within cgo calls, it becomes more troublesome. For debugging cgo programs or memory analysis, I haven’t found good methods myself, and welcome everyone’s suggestions.

Actual Pitfalls Encountered

When running the encapsulated Go SDK in the production environment, we found that all CPU cores were abnormally saturated. Through Go’s CPU pprof, we eventually pinpointed to a single heavy computation cgo call, where approximately hundreds of goroutines were concurrently calling it continuously.

go func() {
	C.heavy_process_func()
}

However, when analyzing process CPU using the perf top command, we discovered that the operation consuming the most CPU time was actually _raw_spin_lock in the kernel layer. Throughout the entire heavy_process_func process, our business logic showed no traces of using spin locks, and the upper-level goroutines calling it had no intersections either. Where did the race condition for mutex locks come from?

Finally, by actively outputting all exception logs from the C++ layer, the truth came to light: it turned out that when C++ throws large amounts of exceptions concurrently across multiple threads, stack unwinding competes for global locks in glibc. Therefore, after properly converging exceptions, the problem was finally resolved. Newer versions of glibc have fixed the issue of multi-threaded exception throwing competing for global locks. Details can be seen in the corresponding issue and patch.

Bug - Concurrently throwing exceptions is not scalable

References

Memory Leak Caused by Using Go STL to Query DB

Fri, 06 Mar 2020 13:54:12 +0800

Problem Origin

Recently, a Go API service was found to have continuously increasing memory usage through regular monitoring. The memory usage rose from the initial 70MB to over 1GB until it ran out of memory and crashed. This clearly indicated a memory leak issue. Since memory leaks rarely occur in languages with built-in garbage collection, if there is one, it must be a significant problem. Therefore, the following detailed investigation process was carried out to accumulate experience for handling similar issues in the future.

Analysis and Investigation

Goroutine Leak

I had previously heard a famous saying: “Nine out of ten memory leaks in Go are due to goroutine leaks.” Is this really the case? Regular queries of runtime.NumGoroutine() can help us determine this, or if pprof is enabled online, we can visit /debug/pprof to see the number of goroutines. I used expvar to periodically expose the total number of goroutines and regularly collected the online data locally.

// server.go
// Expose statistical data via expvar.Handler to the HTTP service
mux := http.NewServeMux()
mux.Handle("/debug/vars", expvar.Handler())


// main.go
func sampleInterval() chan struct{} {
	done := make(chan struct{})
	go func() {
		ticker := time.NewTicker(1 * time.Second)
		numG := expvar.NewInt("runtime.goroutines")
		defer ticker.Stop()

		for {
			select {
			case <-ticker.C:
				numG.Set(int64(runtime.NumGoroutine()))
			case <-done:
				return
			}
		}
	}()
	return done
}

// Run locally
// curl http://host:port/debug/vars
{
	"runtime.goroutines": 48
}

The done channel is returned so that we can gracefully exit when shutting down the system. After some time of data collection and chart analysis, I found that it was not a goroutine leak. If it were, we could view the status of each goroutine online by visiting /debug/pprof/goroutine?debug=1 for further analysis. However, through the process of elimination, this was clearly not the cause of the issue.

Long-term Reference Holding

Now a new question naturally emerges: if the number of goroutines rises and then always manages to fall back, it means that non-resident goroutines can finish normally. We can rule out leaks caused by goroutines being blocked and unable to exit. But can we conclude that memory leaks are completely unrelated to goroutines? The answer is no, because we cannot be sure whether goroutines might cause some of our reference objects to be continuously held. During the Mark phase, these objects that are still being held certainly cannot be garbage collected. If they are being held for some reason and accompanied by new memory allocations, this is the second major culprit leading to continuous memory growth.

So, the problem has shifted to the memory allocation of objects or object groups. This could be a large slice that continuously adds elements while being referenced throughout the object’s lifecycle, or it could be an object holding pointers to other objects, like a linked list reference relationship. If this reference chain keeps growing, it will similarly cause memory leaks.

Using pprof to Sample and Analyze Heap Memory

By obtaining pprof heap profiles, we can directly generate snapshots of the current heap memory allocation online and persist them in the form of .pb.gz compressed packages for further offline analysis.

go tool pprof http://[domain]/debug/pprof/heap

Using the command line, we can persist the heap memory snapshot and various statistics locally for analysis, entering the pprof interactive command line. I directly captured two snapshots from the production environment at different times for analysis:

File: binary
Type: inuse_space
Time: Feb 27, 2020 at 12:06pm (CST)
(pprof) top
Showing nodes accounting for 44174.92kB, 93.50% of 47247.81kB total
Showing top 10 nodes out of 114
      flat  flat%   sum%        cum   cum%
35842.64kB 75.86% 75.86% 35842.64kB 75.86%  myfuncA
 1825.78kB  3.86% 79.73%  1825.78kB  3.86%  github.com/mozillazg/go-pinyin.init
 1805.17kB  3.82% 83.55%  2354.01kB  4.98%  compress/flate.NewWriter
 1024.14kB  2.17% 85.71%  2048.25kB  4.34%  database/sql.(*DB).prepareDC
 1024.08kB  2.17% 87.88%  1024.08kB  2.17%  net.(*file).getLineFromData
  561.50kB  1.19% 89.07%   561.50kB  1.19%  html.populateMaps
  548.84kB  1.16% 90.23%   548.84kB  1.16%  compress/flate.(*compressor).initDeflate
  516.76kB  1.09% 91.32%   516.76kB  1.09%  net/http.(*http2Framer).WriteDataPadded
  513.31kB  1.09% 92.41%   513.31kB  1.09%  regexp.onePassCopy
  512.69kB  1.09% 93.50%   512.69kB  1.09%  vendor/golang.org/x/net/http2/hpack.(*headerFieldTable).addEntry

File: binary
Type: inuse_space
Time: Feb 27, 2020 at 5:58pm (CST)
(pprof) top
Showing nodes accounting for 80.65MB, 83.62% of 96.45MB total
Showing top 10 nodes out of 145
      flat  flat%   sum%        cum   cum%
      35MB 36.29% 36.29%       35MB 36.29%  myfuncA
    8.50MB  8.81% 45.10%       11MB 11.41%  github.com/lib/pq.parseStatementRowDescribe
    7.05MB  7.31% 52.42%     7.05MB  7.31%  database/sql.(*DB).addDepLocked
       6MB  6.22% 58.64%        6MB  6.22%  encoding/json.(*decodeState).literalStore
       5MB  5.18% 63.82%    19.50MB 20.22%  github.com/lib/pq.(*conn).prepareTo
    4.59MB  4.76% 68.58%     4.59MB  4.76%  reflect.unsafe_NewArray
    4.50MB  4.67% 73.25%       11MB 11.41%  queryfuncB
       4MB  4.15% 77.40%    18.56MB 19.24%  database/sql.(*DB).prepareDC
    3.50MB  3.63% 81.03%     3.50MB  3.63%  github.com/lib/pq.decideColumnFormats

Using top, we can view the functions holding the most memory, sorted accordingly. By comparing the functions holding the most memory, I found that although myfuncA held a lot of memory, its memory usage remained unchanged between the two snapshots, while the total memory usage increased from 47MB to 96MB. Therefore, we can rule out myfuncA as the source of the leak. It’s worth noting that not all of the additional 50MB is due to memory leaks; DB query deserialization also requires significant memory allocation. According to Go runtime’s memory allocation strategy, GC does not immediately reclaim this memory, and even if it does, it won’t quickly return idle memory to the OS. This memory will still exist in the memory allocator’s multi-level memory pool for some time. Therefore, there are still some interfering factors hindering our analysis. By the way, regarding the reflect.unsafe_NewArray function, some people previously suspected it might cause memory leaks. In fact, whenever a JSON is deserialized into a slice, it uses this function to allocate memory. Its upper layer is reflect.MakeSlice, which can create a slice based on type information obtained through reflection, essentially a constructor for a generic slice.

Comparing Differences Between Two Snapshots

Simply visually inspecting the differences between the two snapshots still has a lot of interference. It might be clearer if we could use the first snapshot as a baseline and view the diff results with the second snapshot.

go tool pprof -base pprof.binary.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz pprof.binary.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz

File: binary
Type: inuse_space
Time: Feb 27, 2020 at 12:06pm (CST)
(pprof) top
Showing nodes accounting for 37.65MB, 76.19% of 49.42MB total
Dropped 1 node (cum <= 0.25MB)
Showing top 10 nodes out of 118
      flat  flat%   sum%        cum   cum%
    6.50MB 13.16% 13.16%        9MB 18.21%  github.com/lib/pq.parseStatementRowDescribe
    5.05MB 10.23% 23.38%     5.05MB 10.23%  database/sql.(*DB).addDepLocked
       5MB 10.12% 33.50%        5MB 10.12%  encoding/json.(*decodeState).literalStore
    4.58MB  9.28% 42.78%     4.58MB  9.28%  reflect.unsafe_NewArray
       4MB  8.10% 50.87%        8MB 16.19%  funcB (callByA)
       4MB  8.10% 58.97%       16MB 32.38%  github.com/lib/pq.(*conn).prepareTo
       3MB  6.07% 65.04%        3MB  6.07%  github.com/lib/pq.decideColumnFormats
    2.50MB  5.06% 70.10%     2.50MB  5.06%  github.com/lib/pq.(*readBuf).string
    1.51MB  3.05% 73.15%     1.51MB  3.05%  github.com/lib/pq.textDecode
    1.50MB  3.04% 76.19%    39.10MB 79.12%  ctrlfuncA

By specifying the baseline with -base and using top again, we can accurately list what makes up the extra 50MB. Categorizing these listed methods: funcA is the controller function, and funcB is the M-layer method called by A that directly interacts with PostgreSQL to obtain data. The rest includes standard library functions for deserializing JSON data fields and functions related to database connections. Initially, we can be certain that the memory leak exists somewhere in this method call chain.

In the interactive command line, using list methodName allows us to analyze the specific usage situations step by step along the method call chain. Excluding the interference of memory allocation for deserialized objects and combining with the occupancy situation of snapshot 2, we can locate the *DB and *conn objects, with method calls pointing to the Preparex method provided by sqlx and the Get method of the Stmt object produced by Preparex.

func (r repoXXX) FindByID(id string) (*T, err) {
	stmt, err := r.Preparex(`MY SQL QUERY`) // ???
	if err != nil {
		return nil, err
	}

	t := new(T)
	err = stmt.Get(t, id) // ???
	if err != nil {
		if err == sql.ErrNoRows {
			return nil, errno.TorNotFound
		}
		return nil, err
	}
	return t, nil
}

Finding the Culprit

sqlx is just a simple wrapper library for database/sql. To understand what these two methods actually do, we need to start from database/sql.

DB Creation

func Open(driverName, dataSourceName string) (*DB, error) {
	// Check if the database driver is already initialized
	driversMu.RLock()
	driveri, ok := drivers[driverName]
	driversMu.RUnlock()
	// Not initialized, report an error
	if !ok {
		return nil, fmt.Errorf("sql: unknown driver %q (forgotten import?)", driverName)
	}

	if driverCtx, ok := driveri.(driver.DriverContext); ok {
		// Pass configuration information for connecting to the database via URI
		connector, err := driverCtx.OpenConnector(dataSourceName)
		if err != nil {
			return nil, err
		}
		// Initialize the DB object and return
		return OpenDB(connector), nil
	}

	// Default connector
	return OpenDB(dsnConnector{dsn: dataSourceName, driver: driveri}), nil
}

OpenDB() is obviously the DB constructor, let’s continue

func OpenDB(c driver.Connector) *DB {
	ctx, cancel := context.WithCancel(context.Background())
	db := &DB{
		connector:    c,
		openerCh:     make(chan struct{}, connectionRequestQueueSize),
		resetterCh:   make(chan *driverConn, 50),
		lastPut:      make(map[*driverConn]string),
		connRequests: make(map[uint64]chan connRequest),
		stop:         cancel,
	}

	go db.connectionOpener(ctx)
	go db.connectionResetter(ctx)

	return db
}

Several key components: openerCh is used to cache requests for creating new connections, with a single purpose and following a first-in-first-out logic, defined as chan struct which is perfect for this use case

resetterCh is used to asynchronously reset the connection’s context and return the current connection’s query execution error

Two goroutines asynchronously execute the above two operations

lastPut is used for debugging, we’ll skip it

connRequests is used for asynchronous request queuing when there are no idle database connections and no new connections can be created. The key is incremental, and the value uses a channel to prevent blocking when reading. The length can easily be used for counting.

func (db *DB) putConnDBLocked(dc *driverConn, err error) bool {
	// ...
	// Discovers that there are requests in the queue
	if c := len(db.connRequests); c > 0 {
		var req chan connRequest
		var reqKey uint64
		for reqKey, req = range db.connRequests {
			break
		}
		delete(db.connRequests, reqKey) // Remove from the queue
		if err == nil {
			dc.inUse = true
		}
		// Allocate a connection for this request
		req <- connRequest{
			conn: dc, // *driverConn is the database connection
			err:  err,
		}
		return true
	}

}

So far, we know that connRequest represents a request to create a new connection, and connRequests is used for global request queuing. Whenever there’s a request to create a new connection, DB tries to reuse available driverConn to satisfy these requests. Note that these are not query requests, but requests to create connections.

Recording Resource Dependencies

The DB object’s conn method is responsible for obtaining connections. Once we have a connection, we can send our statements to the database. Taking a SELECT statement query as an example:

func (db *DB) query(ctx context.Context, query string, args []interface{}, strategy connReuseStrategy) (*Rows, error) {
	// Get a connection or return an error if it fails
	dc, err := db.conn(ctx, strategy)
	if err != nil {
		return nil, err
	}

	// The familiar stmt, Rows object is created here, bound to driverConn execution and then returned
	return db.queryDC(ctx, nil, dc, dc.releaseConn, query, args)
}

Let’s not go too far ahead, as we’re not using this method to access the database. But the above call can help clarify how the standard library organizes these underlying objects. Now, let’s get back to sqlx’s preparex method, which is where our problem begins. It ultimately calls the PrepareContext method of the standard library’s DB:

func (db *DB) PrepareContext(ctx context.Context, query string) (*Stmt, error) {
	var stmt *Stmt
	var err error
	// Set a retry count for failures, use a flag to control the connection acquisition strategy (prioritize using cached connections)
	for i := 0; i < maxBadConnRetries; i++ {
		stmt, err = db.prepare(ctx, query, cachedOrNewConn)
		if err != driver.ErrBadConn {
			break
		}
	}
	if err == driver.ErrBadConn {
		return db.prepare(ctx, query, alwaysNewConn)
	}
	return stmt, err
}

func (db *DB) prepare(ctx context.Context, query string, strategy connReuseStrategy) (*Stmt, error) {
	// Similarly tries to get a connection or returns an error if it fails
	dc, err := db.conn(ctx, strategy)
	if err != nil {
		return nil, err
	}
	return db.prepareDC(ctx, dc, dc.releaseConn, nil, query)
}

The prepare method here is very similar to the query method mentioned earlier. We’ll discuss their differences later. For now, let’s continue following the prepareDC clue.

// cg being nil means the connection pool is enabled! (emphasizing this point)
func (db *DB) prepareDC(ctx context.Context, dc *driverConn, release func(error), cg stmtConnGrabber, query string) (*Stmt, error) {
	var ds *driverStmt
	var err error
	defer func() {
		release(err)
	}()
	withLock(dc, func() {
		ds, err = dc.prepareLocked(ctx, cg, query)
	})
	if err != nil {
		return nil, err
	}
	stmt := &Stmt{
		db:    db,
		query: query,
		cg:    cg,
		cgds:  ds,
	}

	// This part is the real purpose of preparex, using the connection pool
	if cg == nil {
		// Prepare saves the obtained connection and statement in advance
		stmt.css = []connStmt{{dc, ds}}
		// Records the total number of currently closed connections, used when cleaning up closed connections
		stmt.lastNumClosed = atomic.LoadUint64(&db.numClosed)
		// Adds dependency reference counting (not clear what it's for at this point), but stmt is definitely being held
		db.addDep(stmt, stmt)
	}
	return stmt, nil
}

The returned stmt is the product of Preparex, which then executes Get. After sqlx’s wrapper, it ultimately calls the standard library’s func (s *Stmt) QueryContext and sqlx’s func (r *Row) scanAny.

It’s certain that after calling Preparex, the stmt reference is held by db. As long as the db object exists, for stmt to be released, there must be corresponding code to manually remove the reference to stmt. Let’s briefly look at QueryContext to see if there’s an operation similar to removeDep that we’re looking for:

func (s *Stmt) QueryContext(ctx context.Context, args ...interface{}) (*Rows, error) {
	// ...
	rowsi, err = rowsiFromStatement(ctx, dc.ci, ds, args...)
	if err == nil {
		rows := &Rows{
			dc:    dc,
			rowsi: rowsi,
		}
		s.db.addDep(s, rows) // HERE

		rows.releaseConn = func(err error) {
			releaseConn(err)
			s.db.removeDep(s, rows) // HERE
		}
		var txctx context.Context
		if s.cg != nil {
			txctx = s.cg.txCtx()
		}
		rows.initContextClose(ctx, txctx)
		return rows, nil
	}

	releaseConn(err)
}

Yes, but these are codes for recording references to rows and reusing database connections, not for releasing the stmt object. Similarly, looking at sqlx’s scanAny, it mainly serializes the data returned by the database to specific fields in the struct passed to Get by calling Scan.

The above findings confirm the culprit of this memory leak: db.dep, a dictionary used for reference counting to ensure that dependencies are released first. This dictionary ensures that only after the element’s dependencies are released can the current element be released, calling the current element’s finalClose method to release the resources held by the current element.

Solving the Leak

Therefore, to solve this memory leak, we must make dep release its reference to stmt. Through a simple search of the sql.go file, the release of the stmt object is hidden in its close method.

func (s *Stmt) Close() error {
	// ...
	if s.cg == nil {
		return s.db.removeDep(s, s) // Actually releases stmt, no longer holding its reference
	}
}

One line of code can solve it:

stmt, _ := r.Preparex(`MY SQL QUERY`)
// This line is essential!!!
defer stmt.Close()
// ...

After redeployment and sampling the memory usage, the results are shown in the following figure:

GC can now properly reclaim the stmt object. Problem solved!

Conclusion

Lessons learned:

Early stage: Perform load tests, solve issues offline if possible; make good use of pprof; write good documentation to avoid others falling into the same trap
Production: Ensure monitoring and alerting are in place, memory usage exceeding a certain threshold must be taken seriously, make good use of http/pprof
Regular practices: Become more familiar with frequently used packages and standard libraries, especially the standard library, as it hides many details that can easily lead to issues if overlooked

Other Potential Leaks (Bonus)

Honestly, this is not the first memory leak I’ve encountered in Go, but Go indeed doesn’t easily have such issues. Through reflection on my multiple encounters with such issues, I’ve found that some goroutine-caused memory leaks can be quickly resolved through pprof or monitoring. Instead, it’s the long-term held references that are easily overlooked. These leak signs often appear in:

Manually maintained object pools
Manually controlled system resources: such as timers, connections, IO buffer streams, etc.
Numerical limits with no default upper bound: such as db.maxOpen

Understanding the Impact of CPU Cache on Concurrent Performance

Mon, 30 Dec 2019 10:42:32 +0800

Generally, each CPU core has L1 and L2 caches, while L3 is a shared cache Stored in cache line units, typically 64 bytes per line

This type of cache utilizing locality principles is the main reason why array access is faster than linked list access

False Sharing Problem

If two cores both have the same cache line, but they operate on different variables within that cache line, a false sharing problem occurs. The processor uses flags to mark the modification status of cache lines and needs to transmit modified cache lines through MESI protocol communication. On the surface, these two variables are modified independently in different L1 caches, but in fact they share the same cache line in L3, resulting in significant communication overhead for data synchronization.

Prerequisites for communication to acquire cache:

Thread work moves from one core to another (related to bound M in GPM model, which is also why it’s better for the originally bound M to run G - utilizing processor local cache!)
Two different cores need to process the same cache line

Solving False Sharing Problems

Example: When multiple threads need to process multiple adjacent data items? Such as: concurrent modification of arrays, adjacent fields in structures being concurrently read and modified, etc.

One feasible approach is boundary padding. In Go, you can typically use anonymous underscore []byte for padding, ensuring that affected data items are located in different cache lines. This breaks sharing and avoids the communication overhead of data synchronization.

Common Go Concurrency Patterns

Sun, 22 Sep 2019 10:08:00 +0000

Go’s concurrency patterns are among its most distinctive programming paradigms. This article summarizes the most common concurrency patterns in Go development, including Request-Response pattern, Single State Manager pattern, Producer-Consumer pattern, Pipeline pattern, and more. These patterns help us better organize and manage goroutines, making concurrent programs more robust and elegant. Through learning these patterns, you will be able to better master Go’s concurrency features and write higher quality concurrent code.

Minimal Request-Response Pattern

// The request function returns a receiver based on parameters
func Glob(pattern string) <-chan Item {
	c := make(chan Item)	// 1. Initialize receiver

	// 2. Anonymous goroutine closure executes program
	go func() {
		defer close(c)
		for name, item := range items {
			if ok, _ := filepath.Match(pattern, name); !ok {
				continue
			}
			c <- item
		}
	}()

	// 3. Quickly return receiver so subsequent programs can read from chan
	return c
}

func main() {
	for item := range Glob("[ab]*") {
		fmt.Println(item)
	}
}

Single State Manager

One goroutine holds a global state, other goroutines can access the state held by this goroutine through channels

reads := make(chan *readOp)
writes := make(chan *writeOp)

// State holder
go func() {
    var state = make(map[int]int)
    for {
        select {
        case read := <-reads:
			// Return result
            read.resp <- state[read.key]
        case write := <-writes:
            state[write.key] = write.val
			// Return result
            write.resp <- true
        }
    }
}()

// Requester
go func() {
    for {
        read := &readOp{
            key:  rand.Intn(5),
            resp: make(chan int)}
        reads <- read
        <-read.resp
     }
}()

go func() {
    for {
        write := &writeOp{
            key:  rand.Intn(5),
            val:  rand.Intn(100),
            resp: make(chan bool)}
        writes <- write
        <-write.resp
     }
}()

Producer First, Consumer Later

// Produce first, known number of productions
chanOwner := func() <-chan int {
    results := make(chan int, 5)
    go func() {
        defer close(results)
        for i := 0; i <= 5; i++ {
            results <- i
        }
    }()
    return results
}

consumer := func(results <-chan int) {
    for result := range results {
        fmt.Printf("Received: %d\n", result)
    }
    fmt.Println("Done receiving!")
}

results := chanOwner()
// Consume
consumer(results)

Consumer Notifies Producer

newRandStream := func(done <-chan interface{}) <-chan int {
    randStream := make(chan int)
    go func() {
        defer fmt.Println("newRandStream closure exited.")
        defer close(randStream)
        for {
            select {
            case randStream <- rand.Int():
            case <-done:
				// Stop production, exit
                return
            }
        }
    }()

    return randStream
}

done := make(chan interface{})
randStream := newRandStream(done)
fmt.Println("3 random ints:")
for i := 1; i <= 3; i++ {
    fmt.Printf("%d: %d\n", i, <-randStream)
}
// Consumer actively requests production to stop
close(done)

Multi-channel Merge for Safe Exit (OR channel)

var or func(channels ...<-chan interface{}) <-chan interface{}
or = func(channels ...<-chan interface{}) <-chan interface{} { 1
    switch len(channels) {
    case 0: 2
        return nil
    case 1: 3
        return channels[0]
    }

    orDone := make(chan interface{})
    go func() {
        defer close(orDone)

        switch len(channels) {
        case 2:
            select {
            case <-channels[0]:
            case <-channels[1]:
            }
        default:
            select {
            case <-channels[0]:
            case <-channels[1]:
            case <-channels[2]:
            case <-or(append(channels[3:], orDone)...):
            }
        }
    }()
    return orDone
}

sig := func(after time.Duration) <-chan interface{}{
    c := make(chan interface{})
    go func() {
        defer close(c)
        time.Sleep(after)
    }()
    return c
}

start := time.Now()
<-or(
    sig(2*time.Hour),
    sig(5*time.Minute),
    sig(1*time.Second),
    sig(1*time.Hour),
    sig(1*time.Minute),
)
fmt.Printf("done after %v", time.Since(start))

As long as one of the channels exits, all other channels exit

When a function needs to be executed concurrently, pack return values into structs and pass them via channels

For example: concurrent http.Get

// Unify return values into result struct
type Result struct {
    Error error
    Response *http.Response
}

checkStatus := func(done <-chan interface{}, urls ...string) <-chan Result {
    results := make(chan Result)
    go func() {
        defer close(results)

        for _, url := range urls {
            var result Result
            resp, err := http.Get(url)
            result = Result{Error: err, Response: resp}
            select {
            case <-done:
                return
            case results <- result:
            }
        }
    }()
    return results
}

done := make(chan interface{})
defer close(done)

urls := []string{"https://www.google.com", "https://badhost"}
for result := range checkStatus(done, urls...) {
    if result.Error != nil {
        fmt.Printf("error: %v", result.Error)
        continue
    }
    fmt.Printf("Response: %v\n", result.Response.Status)
}

At the same time, this pattern inspires us to introduce an observer of program state for fault tolerance, which can control the direction of the program simultaneously. In the above example, we default that observer to be main

Error Control

done := make(chan interface{})
defer close(done)

errCount := 0
urls := []string{"a", "https://www.google.com", "b", "c", "d"}
for result := range checkStatus(done, urls...) {
    if result.Error != nil {
        fmt.Printf("error: %v\n", result.Error)
        errCount++
        if errCount >= 3 {
			// When there are too many errors, break and terminate request goroutine
            fmt.Println("Too many errors, breaking!")
            break
        }
        continue
    }
    fmt.Printf("Response: %v\n", result.Response.Status)
}

Pipeline Pattern

Break down the production process, decouple different stages, multiple goroutines collaborate while remaining independent from each other Prerequisite for decomposition: Different stages process data of the same type

generator := func(done <-chan interface{}, integers ...int) <-chan int {
    intStream := make(chan int)
    go func() {
        defer close(intStream)
        for _, i := range integers {
            select {
            case <-done:
                return
            case intStream <- i:
            }
        }
    }()
    return intStream
}

multiply := func(
  done <-chan interface{},
  intStream <-chan int,
  multiplier int,
) <-chan int {
    multipliedStream := make(chan int)
    go func() {
        defer close(multipliedStream)
        for i := range intStream {
            select {
            case <-done:
                return
            case multipliedStream <- i*multiplier:
            }
        }
    }()
    return multipliedStream
}

add := func(
  done <-chan interface{},
  intStream <-chan int,
  additive int,
) <-chan int {
    addedStream := make(chan int)
    go func() {
        defer close(addedStream)
        for i := range intStream {
            select {
            case <-done:
                return
            case addedStream <- i+additive:
            }
        }
    }()
    return addedStream
}

// Each stage can safely exit goroutine using done channel
done := make(chan interface{})
defer close(done)

// Each stage returns a channel of the data type to be processed
intStream := generator(done, 1, 2, 3, 4)
pipeline := multiply(done, add(done, multiply(done, intStream, 2), 1), 2)

for v := range pipeline {
    fmt.Println(v)
}

Multiple goroutines connected together through multiple channels of the same type and sharing a done channel

Generator Pattern

Infinite repetitive generation

repeat := func(
    done <-chan interface{},
    values ...interface{},
) <-chan interface{} {
    valueStream := make(chan interface{})
    go func() {
        defer close(valueStream)
		// Infinite loop repeatedly sends slice content until stop signal is received
        for {
            for _, v := range values {
                select {
                case <-done:
                    return
                case valueStream <- v:
                }
            }
        }
    }()
    return valueStream
}

The above pattern can generate a stream. Let’s now read from this stream.

Read on-demand from stream

take := func(
    done <-chan interface{},
    valueStream <-chan interface{},
    num int,
) <-chan interface{} {
    takeStream := make(chan interface{})
    go func() {
        defer close(takeStream)
        for i := 0; i < num; i++ {
            select {
            case <-done:
                return
            case takeStream <- <- valueStream:
            }
        }
    }()
    return takeStream
}

done := make(chan interface{})
defer close(done)

// Take first 10 elements from stream then exit
for num := range take(done, repeat(done, 1), 10) {
    fmt.Printf("%v ", num)
}

Worker Pool Pattern

worker := 8
c := make(chan int, l)

var wg sync.WaitGroup
wg.Add(worker)

for i:=0;i<worker;i++ {
	go func(){
		for row := range c {
			for col := range m.m[row] {
				fillPixel(m, row, col)
			}
		}
	}()
}

for row := range m.m {
	c <- row
}
close(c)

Pub-Sub Pattern

Each subscriber holds a channel Each subscriber has a filter for content filtering Filters are typically boolean function types Publisher uses a map to store mappings from subscriber channels to filter functions During each publish, iterate through the map and deliver content after filtering with corresponding filters

Event-Driven Pattern

Often used to trigger other operations during the execution of certain operations such as notifications These event handling operations need to be registered in advance, then triggered at appropriate times

package events

import (
    "time"
)

var UserCreated userCreated

// Define context payload
type UserCreatedPayload struct {
    Email string
    Time  time.Time
}

// Function chain with payload context
type userCreated struct {
    handlers []interface{ Handle(UserCreatedPayload) }
}

// Register adds event handler functions to slice
func (u *userCreated) Register(handler interface{ Handle(UserCreatedPayload) }) {
    u.handlers = append(u.handlers, handler)
}

// Trigger sequentially triggers goroutines to execute context-aware handler functions
func (u userCreated) Trigger(payload UserCreatedPayload) {
    for _, handler := range u.handlers {
        go handler.Handle(payload)
    }
}

package main

import (
    "time"

    "github.com/stephenafamo/demo/events"
)

func init() {
    createNotifier := userCreatedNotifier{
        adminEmail: "the.boss@example.com",
        slackHook: "https://hooks.slack.com/services/...",
    }

	// Register handler callbacks
    events.UserCreated.Register(createNotifier)
}

type userCreatedNotifier struct{
    adminEmail string
    slackHook string
}

func (u userCreatedNotifier) notifyAdmin(email string, time time.Time) {
    // ...
}

func (u userCreatedNotifier) sendToSlack(email string, time time.Time) {
    // ...
}

// As long as it matches the handler function signature, internal implementation can be customized
func (u userCreatedNotifier) Handle(payload events.UserCreatedPayload) {
    // Do something with our payload
    u.notifyAdmin(payload.Email, payload.Time)
    u.sendToSlack(payload.Email, payload.Time)
}

Triggering event handler functions

package auth

import (
    "time"

    "github.com/stephenafamo/demo/events"
    // Other imported packages
)

func CreateUser() {
    // ...
	// After decoupling, only need to pass required payload, no need to care about specific processing logic
    events.UserCreated.Trigger(events.UserCreatedPayload{
        Email: "new.user@example.com",
        Time: time.Now(),
    })
    // ...
}

Leetcode 989. Add to Array-Form of Integer

Thu, 14 Feb 2019 00:00:00 +0000

Problem Description

For a non-negative integer X, the array-form of X is an array of its digits in left to right order. For example, if X = 1231, then the array form is [1,2,3,1]

Given the array-form A of a non-negative integer X, return the array-form of the integer X+K.

// Author: Tecker
// date: 2019.2.14
// 132ms, 13.3MB beat 99.19%, 100.00%
// time: O(N), space: O(1) or O(len(K))
class Solution {
public:
    vector<int> addToArrayForm(vector<int>& A, int K) {
        // If the number represented by A is smaller than K (K max 10000)
        if (A.size()<5) {
            int t=1;
            int sum=0;
            for(int i=A.size()-1;i>=0;--i) {
                sum+=A[i]*t;
                t*=10;
            }
            // Direct conversion and addition, then modify A in place and return
            sum+=K;
            if (sum==0) return vector<int>{0};

            // First insert from low to high digits at the back, then reverse
            int pos=0;
            while(sum!=0) {
                if (pos>=A.size())
                    A.push_back(sum%10);
                else
                    A[pos]=sum%10;
                sum/=10;
                ++pos;
            }

            reverse(A.begin(), A.end());
            return A;
        }

        // Directly add K to the lowest position, then propagate to higher positions
        bool needMore = false;
        for(int i=A.size()-1;i>=0;--i) {
            A[i]+=K;
            if (A[i] < 10) break;
            if (i==0) needMore=true;
            K=A[i]/10;
            A[i]%=10;
        }

        if (!needMore) return A;

        // Need to prepend 1
        reverse(A.begin(), A.end());
        A.push_back(1);
        reverse(A.begin(), A.end());
        return A;
    }
};

Key Insights

To insert elements at the beginning of a vector in place, you can first reverse it, add to the end, then reverse again.

Leetcode 985. Sum of Even Numbers After Queries

Wed, 13 Feb 2019 00:00:00 +0000

Problem Description

Time Complexity: O(N)

Space Complexity: O(1)

// Author: Tecker
// 176ms, 28.7MB; beat 96.40%, 100%
class Solution {
public:
    vector<int> sumEvenAfterQueries(vector<int>& A, vector<vector<int> >& queries) {
        vector<int> res(A.size(), 0);
        int sum=0;
        A[queries[0][1]]+=queries[0][0];
        for(int &num : A) {
            if (num%2==0) sum+=num;
        }
        res[0]=sum;
        for(int i=1;i<queries.size();++i) {
            int val=queries[i][0];
            int idx=queries[i][1];
            int tmp = A[idx];
            A[idx]+=val;
            if (isEven(tmp)) {
                if (!isEven(A[idx]))
                    sum-=tmp;
                else
                    sum+=val;
            } else {
                if (isEven(A[idx])) sum+=A[idx];
            }
            res[i]=sum;
        }
        return res;
    }

private:
    inline bool isEven(int a) {
        return !(a & 1);
    }
};

Initially I wanted to use modulo operations, but accidentally fell into a big pit. Need to pay attention to the correct calculation method for negative number modulo in the future.

int mod(int x, int m) {
    return (x%m + m)%m;
}

Insight: When performing modulo operations on negative numbers in different languages, the results may be different.

POJ 3616 Milking Time

Sun, 29 Jul 2018 00:00:00 +0000

Original Problem Link

Knowledge Point: Weighted Interval DP

Solution Report

#include <cstdio>
#include <algorithm>
#include <iostream>
#include <vector>

using namespace std;

struct P {
  int start, end, e;
};

bool cmp(const P &a, const P &b) {
  return a.start < b.start;
}

int N, M, R;
struct P a[1002];
int dp[1002];

int main() {
  scanf("%d %d %d", &N, &M, &R);
  int i, j;
  for(i=0;i<M;i++) {
    struct P p;
    scanf("%d %d %d", &p.start, &p.end, &p.e);
    // Convert to weighted interval DP
    p.end += R;
    a[i] = p;
  }

  sort(a, a+M, cmp);
  for(i=0;i<M;++i) {
    dp[i] = a[i].e;
    for(j=0;j<i;++j) {
      // Select all previous time periods allowed by current period and take maximum value
      if (a[j].end <= a[i].start) dp[i] = max(dp[i], dp[j]+a[i].e);
    }
  }

  cout << *max_element(dp, dp+M) << endl;
  return 0;
}

POJ 2385 Apple Catching 0ms Solution

Sat, 14 Jul 2018 00:00:00 +0000

Original Problem Link

Knowledge Point: Dynamic Programming

Solution Report

#include <cstdio>
#include <algorithm>
#include <iostream>

using namespace std;

// Why defining j part as 31 doesn't work? When calculating forward, there are movements of 0, ..., 30 times, totaling 31 states, so it needs to be defined as 32
// d[i+1-th minute][already moved j times][located at tree mov(j)+1] Brute force search for result
int d[1000][32][2];
int T, W;
int apple[1000];

// Since we can directly calculate k when j is known, we can directly remove the unnecessary k-level loop to make the entire algorithm faster
int mov(int a) {
  if ((a & 0x1) == 0) return 0;
  return 1;
}

int main() {
  scanf("%d %d", &T, &W);
  int i, j, k;
  for(i=0;i<T;++i) {
    scanf("%d", &apple[i]);
    apple[i]--;
  }
  if (apple[0] == 0) {
    d[0][0][0] = 1;
    d[0][1][1] = 0;
  } else {
    d[0][0][0] = 0;
    d[0][1][1] = 1;
  }

  for(i=0;i<T-1;++i) {
    for(j=0;j<=W;++j) {
      if (mov(j) == apple[i+1]) {
        d[i+1][j][mov(j)] = max(d[i+1][j][mov(j)], d[i][j][mov(j)] + 1);
        d[i+1][j+1][mov(j+1)] = max(d[i+1][j+1][mov(j+1)], d[i][j][mov(j)]);
      } else {
        d[i+1][j+1][mov(j+1)] = max(d[i+1][j+1][mov(j+1)], d[i][j][mov(j)] + 1);
        d[i+1][j][mov(j)] = max(d[i+1][j][mov(j)], d[i][j][mov(j)]);
      }
    }
  }
  cout << *max_element(d[T-1][0], d[T-1][0] + 2*(W+1)) << endl;
  return 0;
}

POJ 2229 Sumsets

Thu, 12 Jul 2018 00:00:00 +0000

Original Problem Link

Knowledge Point: Dynamic Programming

Solution Report

#include <cstdio>
#include <string.h>

int sum;
int d[1000001];

int main() {
  scanf("%d", &sum);
  int i;
  // Initialize the unique solution count, d[1] = 1 means there is only 1 way to decompose 1
  d[1] = 1;
  for(i=2;i<=sum;++i) {
    // Find the pattern: when i is even, the number of solutions equals the number of solutions for i/2 plus the number of solutions for i-1
    // When i is even, it reflects the sum of powers of 2, because except for adding 1, all other additions are even powers of 2 (2^n), which is an important starting point
    if ((i & 0x1) == 0) {
      d[i] = d[i/2];
    }
    // Regardless of odd or even, add the solution count of the previous number
    d[i] += d[i-1];
    // When getting WA, look at the problem carefully again and realize we need to take modulo
    d[i] %= 1000000000;
  }

  printf("%d\n", d[sum]);

  return 0;
}

Conclusion

Focus on the characteristics of the numbers in the problem.

POJ 3176 Cow Bowling

Wed, 11 Jul 2018 00:00:00 +0000

Original Problem Link

Knowledge Point: Dynamic Programming

Solution Report

#include <cstdio>
#include <algorithm>
#include <iostream>

using namespace std;

int a[350][350];
int d[350][350];
int N;

int main() {
  scanf("%d", &N);
  int i, j;
  for(i=0;i<N;++i) {
    for(j=0;j<i+1;++j) {
      scanf("%d", &a[i][j]);
    }
  }

  d[0][0] = a[0][0];
  for(i=0;i<N-1;++i) {
    for(j=0;j<=i;++j) {
      // Note here the memoization technique that maximizes values for overlapping paths
      d[i+1][j+1] = max(d[i+1][j+1], d[i][j] + a[i+1][j+1]);
      d[i+1][j] = max(d[i+1][j], d[i][j] + a[i+1][j]);
    }
  }

  cout << *max_element(d[N-1], d[N-1]+N) << endl;
  return 0;
}

Poj 3262 Protecting the Flowers

Wed, 11 Jul 2018 00:00:00 +0000

Original Problem Link

Knowledge Point: Greedy Algorithm

Solution Report

#include <cstdio>
#include <iostream>
#include <algorithm>
#include <vector>

using namespace std;

struct P {
  int t, d;
};

// Greedy approach to minimize cost
// cost = 2 * T * (total - D)
// Therefore, larger D and smaller T are better
// Key insight: convert to D / T ratio - larger is better
bool cmp(const P &a, const P &b) {
  return b.d * a.t < a.d * b.t;
}

vector<P> v;

int N;

int main() {
  scanf("%d", &N);
  int i, total;
  total = 0;
  for(i=0;i<N;++i) {
    struct P p;
    scanf("%d %d", &p.t, &p.d);
    v.push_back(p);
    total += p.d;
  }

  sort(v.begin(), v.end(), cmp);  
  unsigned long long res = 0;
  for(i=0;i<N;++i) {
    total -= v[i].d;
    res += total * v[i].t * 2;
  }

  cout << res << endl;
  return 0;
}

POJ 1017 Packets

Tue, 10 Jul 2018 00:00:00 +0000

Topic: Greedy Algorithm

Solution Report

#include <cstdio>
#include <algorithm>

int square[7];

// Solution: Always place the largest items first

int solve() {
  int res = 0;
  // 6x6 directly forms one box
  res += square[6];
  // 5x5 forms one box first, then fill with 1x1
  res += square[5];
  square[1] -= 11 * square[5];
  // 4x4 forms one box first, then fill with 2x2, 1x1
  res += square[4];
  int remain = 20 * square[4];
  while(remain != 0) {
    if (square[2] > 0) {
      square[2]--;
      remain-=4;
    } else if (square[1] > 0) {
      square[1]--;
      remain--;
    } else {
      break;
    }
  }

  // 3x3 priority: use 3x3 to form boxes first, then 2x2, 1x1
  res += square[3] / 4;
  square[3] %= 4;
  if (square[3] > 0) {
    res++;
    int a;
    // Trap: 2x2 has quantity restrictions when filling! Handle by cases
    switch(square[3]) {
      case 1:
      a = 5;
      remain = 27;
      break;
      case 2:
      a = 3;
      remain = 18;
      break;
      case 3:
      a = 1;
      remain = 9;
      break;
    }
    while(remain && a && square[2] > 0) {
      remain-=4;
      square[2]--;
      a--;
    }
    square[1]-=remain;
  }

  // If 2x2 still remains, form boxes by themselves, remaining ones continue to combine with 1x1
  if (square[2] > 0) {
    res += square[2] / 9;
    square[2] %=9;
    if (square[2] > 0) {
      res++;
      square[1] -= 36 - 4 * square[2];
    }
  }
  // If 1x1 still remains, can only form boxes by themselves
  if (square[1] > 0) {
    res += square[1] / 36;
    if (square[1] % 36) res++;
  }
  return res;
}

int main() {
  int i;
  while(1) {
    for(i=1;i<=6;i++) {
      scanf("%d", &square[i]);
    }
    if (square[1]+square[2]+square[3]+square[4]+square[5]+square[6] == 0) {
      break;
    }
    printf("%d\n", solve());
  }
  return 0;
}

poj 3040 Allowance

Tue, 10 Jul 2018 00:00:00 +0000

Knowledge Points:

Greedy Algorithm

Solution Report

#include <cstdio>
#include <iostream>
#include <algorithm>
#include <vector>
#include <string.h>
#include <limits>

using namespace std;

struct Coin {
  int d;
  int b;
};

bool cmp(const Coin &a, const Coin &b) {
  return a.d > b.d;
}

vector<Coin> v;
int N, C;
int need_count[20];

int main() {
  scanf("%d %d", &N, &C);
  int res = 0;
  for(int i=0;i<N;i++) {
    struct Coin c;
    scanf("%d %d", &c.d, &c.b);
    if (c.d >= C) {
      res += c.b;
      continue;
    }
    v.push_back(c);
  }

  sort(v.begin(), v.end(), cmp);
  N = v.size();

  int i, j;
  while(1) {
    int sum = C;
    memset(need_count, 0, sizeof(need_count));
    // Greedily take from largest to smallest until exactly sum or close to sum
    for(i=0;i<N;++i) {
      if (sum > 0 && v[i].b > 0) {
        int has = min(v[i].b, sum / v[i].d);
        if (has > 0) {
          sum -= has * v[i].d;
          need_count[i] = has;
        }
      }
    }

    // Greedily take from smallest to largest to reach sum
    for(i=N-1;i>=0;--i) {
      if (sum > 0 && v[i].b > 0) {
        // Allow exceeding one denomination value to make up sum
        int has = min(v[i].b - need_count[i], (sum+v[i].d-1) / v[i].d);
        if (has > 0) {
          need_count[i] += has;
          sum -= has * v[i].d;
        }
      }
    }

    if (sum > 0) break;

    int weeks = numeric_limits<int>::max();
    for(i=0;i<N;++i) {
      if (need_count[i] == 0) continue;
      // Number of times the optimal solution for C can be satisfied, which is weeks
      weeks = min(weeks, v[i].b / need_count[i]);
    }
    res += weeks;
    // Finally update remaining coin counts
    for(i=0;i<N;++i) {
      if (need_count[i] == 0) continue;
      v[i].b -= weeks * need_count[i];
    }
  }
  
  printf("%d\n", res);
  return 0;
}

Conclusion

Classic greedy algorithm problem, focus on learning greedy thinking patterns.

poj 1979 Red and Black

Thu, 28 Jun 2018 00:00:00 +0000

Knowledge Point: DFS

Solution

#include <cstdio>

int W, H;
char tile[25][25];
int pos_x, pos_y;
int sh_x[4] = {0, 0, -1, 1};
int sh_y[4] = {1, -1, 0, 0};
int res = 0;

void solve(int x, int y)
{
  tile[y][x] = '#';
  res++;
  
  for (int i=0;i<4;i++) {
    int new_x = x + sh_x[i];
    int new_y = y + sh_y[i];
    if (new_x >= 0 && new_x <W && new_y >=0 && new_y < H && tile[new_y][new_x] != '#') {
      solve(new_x, new_y);
    }
  }
}

int main()
{

  while (1)
  {
    scanf("%d %d", &W, &H);
    getchar();
    if (!W && !H) {
      break;
    }

    for (int i = 0; i < H; i++) {
      fgets(tile[i], 25, stdin);
      tile[i][W] = '#';
    }

    for (int i=0;i<H;i++) {
      for (int j=0;j<W;j++) {
        if (tile[i][j] == '@') {
          pos_x = j;
          pos_y = i;
        }
        if (i==H-1) {
          tile[H][j] = '#';
        }
      }
    }

    solve(pos_x, pos_y);
    printf("%d\n", res);
    res = 0;
  }
  return 0;
}

LeetCode Solution: Manacher's Algorithm for Longest Palindromic String

Tue, 17 Apr 2018 00:00:00 +0000

Manacher, commonly known as the 马拉车 (Malache) algorithm, is the protagonist of this article. It is an efficient algorithm that can reduce the time complexity of solving the longest palindromic string problem to O(N). When I first encountered problems asking for the longest palindromic string, I initially used the brute force approach - traversing all possible substrings with O(N²) complexity, then using O(N) time to find the maximum palindrome length. Although using a map to store palindrome information for various index pairs could reduce some calculations, this approach clearly had too high time complexity and was prone to timeout issues for certain test cases.

The general idea of this algorithm is as follows:

Simplify odd/even number considerations and boundary interference for palindrome judgment by inserting special symbols
The half of the entire boundary (excluding the center point) of the center point exactly equals the maximum palindromic string length at the center point
Utilize the symmetry rule of maximum palindromic substrings within large palindromic boundaries to greatly simplify calculations

I implemented the above algorithm using Go language. The detailed code is as follows:

// LeetCode No.5 Longest Palindromic Substring
func longestPalindrome(s string) string {
    // Convenient for us to directly use slice operations to get results later
    origin := []byte(s)

    // A single character doesn't need palindrome consideration
    if len(origin) == 1 {
        return s
    }

    var maxCenterIndex int

    bs := process(s)
    p := make([]int, len(bs))

    c := 0
    r := 0

    for i:=1;i<len(bs)-1;i++{
        // mirror is the symmetric index of i with c as the axis of symmetry
        mirror := 2*c-i

        // When there is a right boundary
        if r > i {
            // When the boundary doesn't exceed the maximum palindromic string length of the large palindromic string center, it equals the maximum palindromic string length at the symmetric position
            if r-i > p[mirror] {
                p[i] = p[mirror]
            } else {
                // When the boundary exceeds, the minimum maximum palindromic string length at position i equals the length from i to the boundary
                p[i] = r-i
            }
        }

        // Calculate the maximum palindromic string length at position i. Previous steps of calculating p[i] save a lot of palindrome comparisons
        // Note +1 and -1 to exclude the comparison of a character with itself
        for bs[p[i]+i+1] == bs[i-p[i]-1] {
            p[i]++
        }

        // Recalculate the center when the boundary expands
        if i+p[i] > r {
            r = i+p[i]
            c = i
        }

        // Each round of calculation gets the maximum palindromic string length centered at bs[i], here we maintain the maximum one
        if p[maxCenterIndex] < p[i] {
            maxCenterIndex = i
        }
    }

    mx := p[maxCenterIndex]

    // Calculate the starting position of the source string from the center position of the intermediate string
    startIndex := (maxCenterIndex - 1) / 2 - mx/2
    endIndex := startIndex + mx
    return string(origin[startIndex:endIndex])
}

// Use buffer to efficiently process the original string, aiming to handle problems caused by odd/even numbers and boundaries in the original string, making the algorithm implementation more concise
// eg: input - "abcabc" -> output - "^#a#b#c#a#b#c#$"
// ps: Go 1.10 version already supports stringBuilder
func process(s string) []byte {
    bs := []byte(s)
    var buffer bytes.Buffer
    buffer.WriteString("^")
    for _,b := range bs {
        buffer.WriteString("#")
        buffer.WriteByte(b)
    }
    buffer.WriteString("#$")
    return buffer.Bytes()
}

BUG Caused by Unbuffered Channels

Wed, 11 Apr 2018 00:00:00 +0000

Recently I was working on a toy-level MapReduce implementation that contains a lot of concurrent programming code. A slight mistake could cause unknown blocking states during Map or Reduce operations, preventing the entire system from processing all tasks as planned and writing the output results to files. After extensive debugging, I discovered that the root cause was unfamiliarity with basic usage of unbuffered channels. When dealing with multiple channel collaborations, it’s inevitable to run into pitfalls.

Let’s start with the key point:

Unbuffered channels require both sender and receiver to be ready simultaneously to perform send and receive operations

Show me the code

During the fault tolerance phase of Tasks, we need to re-execute those failed tasks. My approach is to save the execution results of each task in a boolean array, where the index corresponds to each task and the element indicates whether the current Task is completed.

for t, failed := range taskFailed {
    if failed == true {
        wg.Add(1)
        addr := <-registerChan
        go func(i int, workerAddr string, p jobPhase) {
            res := call(workerAddr, "Worker.DoTask", DoTaskArgs{
                JobName:       jobName,
                File:          mapFiles[i],
                Phase:         p,
                TaskNumber:    i,
                NumOtherPhase: n_other,
            }, nil)
            if res == true {
                registerChan <- workerAddr
                wg.Done()
            }
        }(t, addr, phase)
    }
}

wg.Wait()

We traversed the task status array, identified those failed tasks, let idle workers execute them, and then simply waited using WaitGroup.

However, upon careful observation and connecting with the previously mentioned requirement that unbuffered channels need both sides ready to perform operations, this implies that if only one side is performing an operation while the other side is not online, the operation will block the goroutine.

Looking back at these two lines of code:

registerChan <- workerAddr
wg.Done()

Note that there may not necessarily be a corresponding registerChan receiver at this moment, because we only reallocate Workers for failed Tasks. Therefore, if only one Task fails, wg.Wait() will remain blocked forever. So we should swap the order:

wg.Done()
registerChan <- workerAddr

This way, the WaitGroup won’t remain blocked indefinitely.

Distributed System Fundamentals

Fri, 06 Apr 2018 00:00:00 +0000

What is a Distributed System?

A system that uses multiple machines to handle the same problems that a single machine would process

Multi-point storage, multi-point computing

Why Do We Need Distributed System?

To solve the growing demand for storage resources and computational resources, distributing system pressure

Three Major Problems Distributed System Must Address

Storage
Transmission (Communication)
Computation

Goals of Distributed System

Able to handle N-fold growth in problems by purchasing N times more hardware

Characteristics of Distributed System

Scalability

Can be divided into three dimensions:

Scale-out scaling: Increasing the number of nodes should linearly speed up the entire system; increasing data volume should not increase latency
Geographic scaling: Using geographically distributed data centers to reduce user response time, properly handling cross-datacenter response latency
Administrative scaling: Adding more nodes should not increase the management burden of the entire system

Performance

Tasks can be processed quickly (low latency)
High throughput (task processing rate)
Consumes fewer computational resources

These factors require constant trade-offs, where low latency is arguably the most important factor, often related to physical limitations that are difficult to solve with money

Latency: refers to the period from when an event occurs until its impact or visibility is realized

A system where data never changes should not have latency issues

The ideal minimum latency depends on the physical distance information must travel

Availability (Fault Tolerance)

One advantage of Distributed System over single machines is that single machines cannot tolerate faults - they either succeed or fail

Distributed System can accommodate a group of unreliable components while building a reliable system at the top level

Systems without redundancy have availability equal to their components; systems with redundant design can tolerate partial failures and therefore have high availability

Availability = Uptime / (Uptime + Downtime)

Fault tolerance: The ability to continue working in predefined ways when errors occur

In short, we cannot tolerate errors that we did not anticipate

Limitations of Distributed System

Physically:

Number of nodes
Distance between nodes

If the number of nodes increases: it reduces system availability, increases management overhead, and node-to-node communication costs will degrade performance, thereby increasing wait time (related to physical distance)

Performance and availability can be defined by guarantees provided to external systems (reliability). These guarantees are agreements between different services, such as: How fast can data written at one point be read at other points? Can written data be guaranteed to be persisted? Within what timeframe can a computation request return results? What impact will component failure have on the entire system?

Errors are incorrect behavior, exceptions are unknown behavior

System Abstraction and Modeling

Good abstractions make systems easier to understand

Systems that hide details externally are easier to understand, but systems that expose more details can achieve better optimization

An ideal system has both clear expression and meets business needs

Design Essence: Partition and Replicate

For a dataset, it can be divided into multiple smaller portions to be distributed across multiple nodes for parallel processing (partitioning), or it can be replicated or cached to different nodes to reduce the distance between clients and servers and improve fault tolerance.

Partitioning

Dividing data into different and independent subsets addresses growing data volumes, improves processing performance and availability. The basis for segmentation relates to the primary way data is accessed, and must address some issues with independent subsets (such as inefficient access between different subsets)

Replication

Enables new computational resources to be applied to new copies of data, reducing latency while enhancing availability

Replication can enhance scalability, performance, and reliability

Replication can avoid single points of failure, accelerate computation, cache I/O to improve throughput

Problems brought by replication:

Cluster data synchronization (maintaining data consistency)

Consistency model selection is crucial: Strong consistency models allow your programs to be programmed as if dealing with a single piece of data, while weak consistency models offer lower latency and higher availability

References:

http://book.mixu.net/distsys/

Zero-Swap Sorting Google Interview Question

Thu, 29 Mar 2018 00:00:00 +0000

Question:

An array of length n contains numbers 0 to n-1 in random order. Now you can only perform swaps between 0 and other numbers. Please sort this array.

package main

import "fmt"

func main() {
    s := []int{3, 5, 4, 0, 1, 2, 6}
    for _, v := range s {
        fmt.Printf("%d ", v)
    }
    fmt.Printf("\nafter:\n")

    var i int
    // First find 0 and put it at the first position
    for s[i] != 0 {
        i++
    }
    s[0], s[i] = s[i], s[0]

    i = 1

    for {
        // Starting from index 1, swap s[i] with 0 to move number x to position s[x]
        for s[i] != i {
            s[s[i]], s[0] = s[0], s[s[i]]
            s[i], s[s[i]] = s[s[i]], s[i]
            // Remember to swap 0 back to its original position each time
            s[i], s[0] = s[0], s[i]
        }
        if i == len(s)-1 {
            break
        }
        // After sorting s[i], proceed to sort s[i+1]
        i++
    }

    for _, v := range s {
        fmt.Printf("%d ", v)
    }
}

Run Code Online

Practicing TDD Testing APIs in Gin

Mon, 26 Mar 2018 00:00:00 +0000

If this article can make you put down your current courier and start becoming a TDD-first programmer from now on, then the purpose of this article has been achieved.

Why TDD is so important?

The most direct manifestation of TDD’s importance is No guessing and Robustness - everything is within the expectations of testing, ensuring that you have thoroughly considered all scenarios. Rather than directly throwing your code or API interfaces to testers and then continuing modifications based on feedback, this back-and-forth process actually reduces efficiency significantly. When we realize this, why not discover and fix most bugs ourselves through testing beforehand?

Secondly, as system scale continues to grow and new features are constantly added, unit tests help us perform system refactoring. We can locate dead code through code coverage, identify robust code by examining call counts to separate them into packages or modules for incremental compilation, and avoid bloating while accelerating build speeds in addressing system degradation issues. Testing is absolutely indispensable in these improvements.

TDD can affect code quality

Prioritizing testable classes or functions greatly influences your coding style. A TDD-first programmer should first consider: what are the dependencies (parameters) of this function? Object instances, basic types, or interfaces? What kind of function is easy to test? Here we must mention a famous quote from Clean Code:

Functions should do one thing. They should do it well and do it only.

When you’re willing to actively write tests, you increasingly hope these functions are like this,无形之中 telling you the shape of your target function. It sounds abstract, but every word is crucial. Simply put, building an engineering project is usually bottom-up. The more flexible the system, the finer the granularity should be - like building blocks, from the data layer that directly interacts with the database, to the persistence service layer, RESTful-based controller layer, validation and permission control, routing layer, logging management, monitoring, etc. Higher-level operations increasingly depend on lower-level operations, so the requirements for code robustness from top to bottom are higher. From a testing perspective, looking from bottom to top, the finer-grained functions at this level bear greater responsibility, and testing at this level is also the easiest to write because they are exactly the type of function that does only one thing.

TDD decreases BUGs

Consider these two controller functions:

func (ctrl *UserController) Register(user *entity.User) {
    // validate user
    // code ....
    // code ....
    // insert into DB
    // code ....
    // code ....
    // err handle
    // code ....
    // code ....
    // response
}

func (ctrl *UserController) Register(user *entity.User) {
    // ... validate.NewUser(user)
    // err handle ...
    // ... services.AddNewUser(user)
    // err handle ...
    // response ...
}

The first function doesn’t follow Do one thing, making the entire function body very bloated with more potential for bugs; the second controller function is very concise and basically doing one thing - handling errors and responding. If the validation layer and service layer functions are already tested, the possibility of problems in the controller decreases by another order of magnitude.

TDD is actually faster

Talk is cheap, show me the code.

Take an end-to-end test that most backend developers need to write, using a gin framework-built API as an example, POST json and then verify the returned json.

Preparation work:

func toReader(t interface{}) *bytes.Reader {
    b, _ := json.Marshal(t)
    return bytes.NewReader(b)
}

func toStruct(r *httptest.ResponseRecorder, t interface{}) error {
    resp := r.Result()
    body, _ := ioutil.ReadAll(resp.Body)
    return json.Unmarshal(body, t)
}

func makeJSONReq(method, path string, data interface{}) *http.Request {
    req, _ := http.NewRequest(method, path, toReader(data))
    req.Header.Add("Content-Type", "application/json")
    req.Header.Add("Accept", "application/json")
    return req
}

These small utility functions can be encapsulated into test suites or toolkits.

Testing one of the API endpoints:

package api_test

import (
    "bytes"
    "encoding/json"
    "log"
    "net/http"
    "net/http/httptest"
    "os"
    "testing"

    "github.com/stretchr/testify/assert"
    "github.com/yujiahaol68/rossy/app/model/checkpoint"

    "github.com/gin-gonic/gin"

    "github.com/yujiahaol68/rossy/app/database"
    "github.com/yujiahaol68/rossy/app/routers/api"
)

var router = gin.Default()

func setup() error {
    api.Router(router)
    return database.OpenForTest()
}

func TestMain(m *testing.M) {
    err := setup()
    if err != nil {
        log.Fatal(err)
        os.Exit(1)
    }
    defer database.Teardown()

    code := m.Run()
    os.Exit(code)
}

func Test_Source(t *testing.T) {
    t.Log("POST: /api/source")
    check := checkpoint.PostSource{"http://www.infoq.com/cn/feed", 2}

    w := httptest.NewRecorder()
    req := makeJSONReq("POST", "/api/source/", &check)
    router.ServeHTTP(w, req)

    var end endpoint.PostSource
    err := toStruct(w, &end)
    if err != nil {
        t.Fatal(err)
    }

    assert.Equal(t, http.StatusCreated, end.Code)
}

The actual test cases to write are very short. If using VSCode, you can run tests without leaving the editor. For larger systems, you don’t need to wait for all routes to be loaded before testing, significantly reducing startup waiting time.

Splitting the view layer into checkpoint and endpoint layers to represent the struct descriptions for input json and returned json respectively. For complex inputs, you can also consider mocking structs. There are many benefits - please start your TDD journey immediately!

More testing practices can be referenced from my example project.

Any mistakes would be greatly appreciated for correction.

Handling Encoding Issues When Parsing XML Documents in Go

Sat, 10 Mar 2018 00:00:00 +0000

Recently I’ve been working on some RSS parsing tasks, so let me document how to parse XML documents with non-UTF-8 encoding. Here’s the code directly:

package rss_test

import (
    "bytes"
    "encoding/xml"
    "fmt"
    "io"
    "testing"

    "github.com/yujiahaol68/rossy/rss"
    "golang.org/x/net/html/charset"
)

func Test_notUTF8(t *testing.T) {
    r := rss.New()

    // Note: don't use xml.Unmarshal() method, it only works with UTF-8 encoding
    d := xml.NewDecoder(bytes.NewReader([]byte(notUTF8rss)))
    // Set encoding processing function, also works with UTF-8 encoding
    d.CharsetReader = func(s string, reader io.Reader) (io.Reader, error) {
        return charset.NewReader(reader, s)
    }
    err := d.Decode(r)

    if err != nil {
        t.Fatal(err)
    }

    for _, item := range r.ItemList {
        fmt.Printf("* %s\n%s\n", item.Title, item.Link)
    }
}

SQL Fundamentals Quick Summary

Sat, 04 Nov 2017 00:00:00 +0000

Inner Join

Contains only rows with associated relationships

SELECT field1, RTRIM(field2) AS f2
FROM table_one INNER JOIN table_two
ON table_one.table_two_id = table_two.id
GROUP BY field1, f2
HAVING COUNT(table_two.credit) > 2
WHERE condition1 AND condition2 AND field1 LIKE '%@hotmail.com'
ORDER BY field3;

Self Join

Can be used as an alternative to subqueries

SELECT c1.cust_id, c1.cust_name, c1.cust_contact
FROM customers AS c1, customers AS c2
WHERE c1.cust_name = c2.cust_name
AND c2.cust_contact = 'Jim';

Outer Join

Includes rows that have no associated relationships

LEFT refers to selecting all rows from the table on the left side of OUTER JOIN

SELECT customers.cust_id, orders.order_num
FROM customers LEFT OUTER JOIN orders
ON customers.cust_id = orders.cust_id;

UNION

Connects multiple queries, but each query must have the same columns

INSERT INTO

INSERT INTO table1(
    field1,
    field2
)
VALUES('a', 2);

VALUES can be replaced with SELECT to insert retrieved data

SELECT INTO can be used to export data from one table to another table

Table Operations

CREATE TABLE ...
DROP TABLE ...

ALTER TABLE table1
ADD field CHAR(20)
DROP COLUMN deletedField

Views

Used for reusing SQL statements, generally used to simplify complex query statements or filter data

CREATE VIEW cv AS
SELECT ...
WHERE ...

SELECT field1, field2
FROM cv
WHERE ...

- The two where clauses are automatically combined

Transactions

Batch SQL statements either execute completely or none at all, or rollback to a specific statement

Used to ensure data integrity

Transactions can be used to manage insert, delete, and update statements

START TRANSACTION
...
SAVEPOINT delete1
...
ROLLBACK TO delete1
COMMIT;

- You can COMMIT where you want to end, and determine whether an error occurred to perform rollback

Important Concepts

Primary Key Unique identifier for each row, values cannot be reused, NULL is not allowed, users cannot modify it, automatically generated, keyword PRIMARY KEY

Foreign Key Ensures that the referenced value exists in another table’s column associated with this column, otherwise an error will occur, keyword

- In definition
REFERENCE table1 (field1)
- In modification
ALTER TABLE table1
ADD CONSTRAINT
FOREIGN KEY (user_id) REFERENCES user (id)

Unique Constraint Guarantees that values in the column are unique, can contain NULL values, can be modified and updated, uses keyword UNIQUE

Check Constraint Restricts the range of values

ADD CONSTRAINT CHECK ( field > 0 )

Index Pre-sorts appropriately, when searching for this sorted column it will be very fast, primary key data itself is sorted, improves search performance, but reduces insert, delete, and update performance because indexes are dynamically updated during these operations

Keyword CREATE INDEX

Trigger Associated with insert, delete, and update operations on specific tables, constraints have better performance, consider using constraints first before using triggers, can be used to maintain database integrity

Linux Process Memory Management Essentials

Tue, 31 Oct 2017 00:00:00 +0000

Basic Concepts

Kernel: Runs on top of hardware, is the core of the operating system, and runs in memory

Process: All programs running on a computer, managed uniformly by the kernel, these processes compose user space

User Process: User-level processes such as GUI processes, server processes, and command-line processes

The kernel runs in kernel mode, can directly access the processor and memory, and cannot tolerate faults; while user processes run in user mode, where a single process crash will not affect the entire system operation

Thread: A more granular concept than process, generally mentioned processes are single-threaded, multi-threaded processes start faster than multi-process ones, main threads will fork other child threads, and can communicate with each other conveniently while sharing I/O and system resources

Kernel Management of Processes and Memory

Process: The kernel decides which process can use the CPU
Memory: The kernel manages all memory, allocates memory to processes, including shared memory between processes and free memory

Understanding Kernel Process Management

For a single-core CPU, every process takes turns using the CPU. The CPU times each incoming process, pauses when time expires, then switches to kernel mode. The kernel records the state of CPU and memory, to restore the suspended process. The kernel executes tasks from the previous time slice such as disk read/write operations, then selects a ready process to execute. The kernel prepares CPU and memory for the new process (or for previously suspended processes), the kernel tells the CPU the time slice, switches the CPU back to user mode, and then the process continues using the CPU for computation

The process of switching CPU usage rights between processes is called context switching, and the kernel runs during the gaps of context switching

System Calls

fork(): Kernel creates a process copy

exec(): Kernel uses the program in exec(program) to replace the current process

Processes that need kernel assistance all use the approach of first fork then exec(..) to request the kernel

Process Tracking

Common commands:

Process information: top and ps

Process and file information: lsof

Trace system calls: strace

Trace shared library calls: ltrace

Tracing Threads

$ ps m

Or press H under top

Measuring CPU Time

$ top -p pid1

Command file execution CPU timing

$ /usr/bin/time command

Average Load

$ uptime

Indicates CPU busyness level, typically web servers have higher average loads, processes start quickly and end quickly

High average load but slow response usually indicates significant memory performance impact. When memory is insufficient, many processes are in ready state but there isn’t enough memory for executing processes

Memory status can be viewed using: free command

Understanding CPU, Kernel, and Memory Relationships

The CPU converts virtual addresses accessed by processes into actual memory addresses through the MMU (Memory Management Unit). The kernel assists the MMU in dividing memory used by processes into smaller regions called pages. Multiple pages are managed using page tables (a data structure maintained by the kernel). The MMU can map virtual addresses to actual memory addresses through these page tables. Page allocation is demand-based, roughly starting with instructions loaded into pages. If the next instruction of this process is executed when the previously loaded memory pages don’t exist, the kernel will load the required memory pages for the instruction, then let the program resume execution. If the process needs more memory, it similarly requires the kernel to obtain more memory before allocation

Memory Page Errors

Minor page fault: If the required memory page exists in main memory but the MMU cannot find it, usually occurs when a process wants more memory but MMU space is limited
Major page fault: If the required page doesn’t exist in main memory, the kernel will attempt to load it from storage media like disk. Most errors affect system performance, particularly likely when loading code from disk for the first time or when memory is insufficient

Viewing method:

$ /usr/bin/time

Monitoring CPU and Memory

$ vmstat n // Refresh data every n seconds

IO Monitoring

Global

$ iostat

Per process

$ iotop

Linux Essentials: Cron Job Management

Mon, 30 Oct 2017 00:00:00 +0000

Cron allows us to create scheduled tasks in Linux. It is a built-in system service that enables repetitive task execution at specified time intervals by adding custom configurations.

crontab

By executing this command, we can load our crontab file into /var/spool/cron/crontabs.

File format:

13 04 * * * Command

The first five columns from left to right represent minute, hour, day, month, and weekday respectively. The asterisk means every.

Separators:

, represents separated time points, such as: 00 10,12 * * * means 10 AM and 12 PM every day

- represents time range, for example: 00 10-12 * * * means every full hour from 10 AM to 12 PM daily

*/n represents every n units of time, such as: */5 * * * * means every 5 minutes

It’s worth noting that the executable files to be run need to use absolute paths

For example: 13 04 * * * /etc/init.d/smb restart

Common parameters

$ crontab -l // List tasks
$ crontab -r name // Delete task file
$ crontab -e // Modify and install task file, enter editor, save and exit after editing to complete successfully

If you need one-time task scheduling for a specific moment, you can use the at command.

Deep Dive into Koa: Source Code Analysis

Tue, 17 Oct 2017 00:00:00 +0000

Introduction

This article provides a comprehensive, in-depth analysis of the core source code of the popular Koa framework from the ground up. It is suitable for developers who are already proficient in using the Koa framework.

Core Mechanism

Now, let’s start from the beginning to see what Koa actually does internally?

// usage
const app = new Koa()

// source code
constructor() {
	super();

	this.proxy = false;
	this.middleware = [];
	this.subdomainOffset = 2;
	this.env = process.env.NODE_ENV || 'development';
	this.context = Object.create(context);
	this.request = Object.create(request);
	this.response = Object.create(response);
}

This is where everything begins. The instance of the Koa class is born, inheriting from Events, which indicates that its subclasses have the ability to handle asynchronous events. However, how Koa handles them is still unknown for now, so let’s mark it with a question mark. During the instantiation process, we can see that three objects are initialized as instance properties: context, request, and response, along with the familiar array middleware that stores all global middleware.

Koa Class Diagram

// usage 
app.use(async (ctx, next) => {
	// middleware function
});

When the use method is called, after confirming it’s an async function, this function will be appended to the middleware array through a push operation:

use(fn) {
    // type checking...
    this.middleware.push(fn);
    return this;
}

At this point, we already have the processing operations, but koa hasn’t really started running yet.

app.listen(3000);

We can say that when the http server is started, koa can truly begin processing our http requests. So what exactly does this concise call do behind the scenes?

const server = http.createServer(this.callback());
return server.listen(...args);

koa uses Node’s native http package to create an http service. All the secrets are hidden within the callback() method.

// source code
const fn = compose(this.middleware);
...
return fn(ctx).then(handleResponse).catch(onerror);

koa itself depends on the koa-compose module. From the way koa uses fn, it appears that middleware should be encapsulated into an object called fn, which returns a Promise by passing in the context object.

Now, let’s dive deep into the koa-compose module to see what it does to our middleware array.

// source code
return function (context, next) {
  // last called middleware #
  let index = -1
  return dispatch(0)
  function dispatch (i) {
    if (i <= index) return Promise.reject(new Error('next() called multiple times'))
    index = i
    let fn = middleware[i]
    if (i === middleware.length) fn = next
    if (!fn) return Promise.resolve()	// recursion ends
    try {
      return Promise.resolve(fn(context, function next () {
        return dispatch(i + 1)	// recursive call to dispatch
      }))
    } catch (err) {
      return Promise.reject(err)
    }
  }
}

The closure inside looks very familiar, doesn’t it? Let’s compare again:

// usage
app.use(async (ctx, next) => {
    // middleware function
});

Here, Promise.resolve(fn(..)) helps us execute the async middleware functions, and the next function explains why Koa’s middleware execution is recursive. It recursively calls the dispatch function to traverse the array. Meanwhile, all middleware functions share the same ctx. Let’s review the external part again:

const handleRequest = (req, res) => {
  res.statusCode = 404;
  const ctx = this.createContext(req, res);
  const onerror = err => ctx.onerror(err);
  const handleResponse = () => respond(ctx);
  onFinished(res, onerror);
  return fn(ctx).then(handleResponse).catch(onerror);
};

return handleRequest;

The context uses the req and res from Node’s native http listener callback function for further encapsulation. This means that for each http request, koa creates a context and shares it among all global middleware. After all middleware have finished executing, the final data to be returned is unified and passed back to res for response. Therefore, in each middleware, we can obtain the data we need from req through ctx for processing, and finally ctx returns the response body to the native res.

Each request has a unique context object, where all request and response related information is unified.

The createContext method further encapsulates req and res:

// source code
const context = Object.create(this.context); // Create an object that has context's prototype methods, following the same pattern
const request = context.request = Object.create(this.request);
const response = context.response = Object.create(this.response);

context.app = request.app = response.app = this;
context.req = request.req = response.req = req;
context.res = request.res = response.res = res;
request.ctx = response.ctx = context;
request.response = response;
response.request = request;

Following the principle of one request per context, context must exist as a temporary object, with everything encapsulated in one object. Therefore, the three properties app, req, and res were created. But you might wonder, why are app, req, and res also encapsulated in both request and response?

By making them share the same app, req, res, and ctx, it enables the transfer of processing responsibilities. Users accessing from outside only need one ctx to get all the data and methods provided by koa. Then koa continues to divide these responsibilities further, such as request being used to further encapsulate req, and response being used to further encapsulate res. This disperses responsibilities, reduces coupling, and sharing all resources makes the entire context highly cohesive, allowing internal elements to access each other.

// source code
context.state = {};

As we can see, state is specifically responsible for storing individual request states as an empty object, allowing users to manage its contents according to their needs.

// source code
const ctx = this.createContext(req, res);
const onerror = err => ctx.onerror(err);

When this large ctx is created, it immediately mounts an error listener. However, the onerror method is not found in createContext. It should belong to one of the modules’ prototype methods. After some searching, we find it located in the context.js file, which defines all the prototype methods that the default context object possesses.

// application.js
// handleRequest()
const onerror = err => ctx.onerror(err);
onFinished(res, onerror);
return fn(ctx).then(handleResponse).catch(onerror);

If exceptions are thrown during the execution of async functions and the exceptions are not captured internally, koa will uniformly use the catch statement of Promise for error handling. Represented in a diagram, it would look like this:

Koa Request Processing Flow Chart

By now, we should be able to resolve the question we had at the beginning of this article: why inherit from Events. It’s actually to use Node’s built-in event listeners to listen to certain events, such as currently known error, to decouple the functionality of error handling.

Global error handling function:

// source code
// application.js

onerror(err) {
  assert(err instanceof Error, `non-error thrown: ${err}`);

  if (404 == err.status || err.expose) return;
  if (this.silent) return;

  const msg = err.stack || err.toString();
  console.error();
  console.error(msg.replace(/^/gm, '  '));
  console.error();
}

The internal core processing flow of Koa has been sorted out. Now let’s explore the internal modules.

Request Module

A large number of getter and setter methods fully extract useful request-related properties provided by Node’s http, along with some necessary helper functions that won’t be elaborated here. Almost all properties under ctx.req in the API documentation are provided here. It’s worth mentioning the this in this section - the troublesome this. This block of code uses a large number of this.req ways to access Node’s native HTTP request information (since we assign Node’s req to context). When using it externally in koa, we access it through the form ctx.req.propertyName… Yes, I think you’ve noticed too. We need to bind our this to point to ctx.req, so in the context.js module, delegation is used to properly bind the this. The assignment code mentioned earlier that enables these modules to access each other actually reduces this troubles to some extent.

// See delegates package for details
// https://github.com/tj/node-delegates

// Koa context.js
delegate(proto, 'request')
  .method('acceptsLanguages')
  .method('acceptsEncodings')
  .access('query')
  .access('path')
  .access('url')
  .getter('origin')

// Internal core implementation of delegates, rebind this via apply
proto[name] = function(){
  return this[target][name].apply(this[target], arguments);
};

Response Module

The internal structure of response.js is similar to request.js. It’s worth mentioning the writable getter - if the request has received a response, it returns true. The value it reads is this.res.finished. In many error messages, we often encounter errors about attempting to write response headers header repeatedly. Even the source of this error comes from the on-finished package, which can execute a callback to save the request status to res when the response ends.

Response Helper Functions

Koa has a unified response function located at the end of application.js, specifically responsible for handling ctx.body and ctx.status for request responses.

// responses
if (Buffer.isBuffer(body)) return res.end(body);
if ('string' == typeof body) return res.end(body);
if (body instanceof Stream) return body.pipe(res);

// body: json
body = JSON.stringify(body);
if (!res.headersSent) {
  ctx.length = Buffer.byteLength(body);
}
res.end(body);

It’s worth noting that the returned body supports Buffer, string, streams, and most commonly json.

Summary of Common Design Patterns and Their Practice - TypeScript Implementation

Sun, 15 Oct 2017 00:00:00 +0000

If there are any shortcomings or errors, please feel free to provide guidance.

Preface:

Thanks to TypeScript’s powerful and flexible OOP syntax, using classic and reliable OOP design patterns in JavaScript has become a reality. Although some companies have already been using TS for large-scale project development, very few projects truly leverage the advantages of TS’s robust object-oriented syntax. Java-like OOP syntax makes it almost identical to Java design patterns in terms of concepts. This article selects some common and important design patterns for refinement and summary, avoiding discussions about which is better between OOP and functional programming. All specific implementations are written in TypeScript, and there’s an easter egg at the end of the article.

GitHub repository address for all practical code examples in this article, welcome to Star

Significance of using design patterns: Libraries and frameworks cannot help us organize applications into architectures that are easy to understand, maintain, and resilient, so we need design patterns.

Basic Design Principles:

Encapsulate what varies
Favor composition over inheritance
Consider potential changes the system may need in the future and principles to handle changes
Program to interfaces, not implementations
Strive for loose coupling in design between interacting objects
The purpose of loose coupling is to minimize object dependencies and enhance system flexibility
Classes should be open for extension but closed for modification (Open-Closed Principle)
Depend on abstractions, not concrete classes
Dependency Inversion Principle: Variables should not hold references to concrete classes; don’t let classes derive from concrete classes; don’t override implemented methods in base classes (methods shared by subclasses); objects that rarely change can violate this, changing objects can use factories to encapsulate changes
Least Knowledge Principle: Don’t couple too many classes together during design, making as few classes as possible interact with each other
Methods within objects should only call methods of the object itself, methods of objects passed as parameters, methods of any objects created by this method, or methods of components within the object (if calling other methods returns an object, don’t call its methods)
Consider all factors comprehensively before adopting principles
High-level components can decide when low-level components participate, but low-level components cannot directly call high-level components (in some cases they can, but never form circular dependencies)
Assign one responsibility to one class; when designed to support a set of related functions, it achieves high cohesion
All the above principles are not golden rules in engineering; actual situations require continuous balancing and trade-offs

Strategy Pattern

The core step is to delegate different variable behaviors in Class 1 into individual behavior classes. Behavior classes 1, 2, 3… will implement different variants of the same interface and exist as properties in various similar classes of Class 1, making algorithm variations independent of customers using the algorithm.

Advantages:

Can always expand new behaviors
The same class can dynamically change its behavior through setter methods

e.g: Strategy

Observer Pattern

Used to better describe one-to-many relationships between objects. When multiple objects (observers) depend on state changes of one object (subject), the Observer pattern helps us distribute state and send updates to various observers, making them loosely coupled.

Core Steps:

Subject class implements Observable interface, implementing management of observers and judgment of whether to push data in the class. Observer class implements Observer interface, implementing update method for subject class to call. When registering new observers, subject objects store observer objects in observer array, and when pushing data, iterate through array calling update method to pass data into their observer objects.

e.g: Observer

Decorator Pattern

Dynamically assign new responsibilities to existing objects, more flexible than inheritance. It uses inheritance, but only to achieve type matching with superclass without obtaining behavior. Behavior comes from decorators and basic components or combinations with other decorators.

As the domino effect in the design pattern world, each new decorated object not only contains the previous decorator’s state but can also extend its own attributes. Through a common superclass, state commonality is preserved and upward chain calls can be made, similar to subclass recursive calls to parent class. This pattern can well replace inheritance.

Core Steps:

Decorated classes uniformly derive from a super abstract class A to obtain their type. Decorator abstract class B inherits from A and saves all states of A-type objects in constructor. Each new decorator wraps the previous decorated object. After decoration, decorated classes can still be continuously extended.

Disadvantages:

Customer code cannot rely on special types, many small objects appear, excessive use leads to program complexity

e.g: Decorator

Factory Pattern

A good way to create objects and solve coupling. Strictly speaking, this is not a pattern but a coding habit. Encapsulate derived class instantiation in static methods of factory class. Abstract factory methods allow subclasses to decide manufacturing methods themselves. Externally, we call abstract methods with great flexibility, implementation doesn’t need to bind specific classes, which brings another advantage: decoupling superclass and subclass objects’ code. And instantiation can be deferred to subclasses.

Core Points:

Use abstract factory methods to replace methods requiring flexibility. To make products customizable, define product classes as abstract classes providing some default attributes and methods or abstract methods. Derived classes can override or implement them themselves. Different factory classes can hide different product manufacturing details while maintaining the same production process.

e.g: Simple Factory

Abstract Factory

Different factory classes implement the same abstract factory interface to manufacture similar products. Similar products then depend on common product interfaces, allowing customers to use them without caring about specific actual products.

The difference between the two is that factory method delegates object instantiation to subclasses, while abstract factory establishes interface contracts between products and customers. Both can decouple customers from specific products.

e.g: Abstract Factory

Singleton Pattern

Instantiate a unique object, provide a global access point. The simplest among all design patterns but very practical and widely used.

Core Points:

Establish a static private variable as the unique instance. External access to this unique instance through static methods, and set constructor to private.

e.g: Singleton

Command Pattern

Encapsulate requests by introducing intermediate proxy objects, decoupling requesters from executors. Often used in work queues or operation logging of large data structures and transaction processing.

Core Steps:

Specific task execution details encapsulated in execute method implementation of Command interface, then pass instances of classes implementing the interface to clients, which execute by calling their execute method.

e.g: Command

Adapter Pattern

Wrap certain interfaces to achieve different purposes, convert interfaces to make them usable by other objects, change interfaces to meet customer expectations.

Core Points:

Object adapter class implements target interface and calls corresponding methods of adapted objects within it, enabling customers to call different interface objects under unchanged customer code.

e.g: Adapter

Facade Pattern

Simplify interfaces using facade class, encapsulate complex subsystems, expose fewer simpler interfaces, create higher-level functionality. Programming to facade can decouple customers from subsystem components.

Core Points:

When constructing Facade class, aggregate various component objects of subsystem, exposed interfaces provide more direct higher-level operations, conforming to least knowledge principle.

e.g: Facade

Template Method Pattern

This encapsulation method allows subclasses to hook themselves into operations at any time. Template method defines algorithm steps and allows subclasses to provide their own implementation for partial steps.

Core Points:

Extract identical steps and implement them in abstract class, different steps implemented separately in subclasses (existing as abstract methods in parent class). Subclasses achieve respective functions by calling the same template method (functions differ partially but cannot be known from template method alone due to unified abstraction). Hook methods can be added to template methods, providing default implementation or left empty for subclasses to override. Through conditional flow control, we can greatly increase algorithm flexibility. Hook methods are usually optional parts of algorithms.

Common Application Scenarios:

When sorting objects, objects need to implement a comparison method first.

Summary (Easter Egg)

If you still find these numerous patterns dizzying and hard to apply at this point, please take these solid practical tips:

Any problem in computer science field can be solved by adding an intermediate layer. Almost all design patterns follow this principle to handle changes, increase flexibility, or perform extensions.
Abstraction and reuse are core requirements. everyone works tirelessly to satisfy these two points.
No perfect pattern can perfectly handle all changes.

Finally, I hope you can apply patterns more to business or framework writing, striving to write clean, extensible, and maintainable architectures.

Reference Materials: Head First Design Patterns Chinese Edition, First Edition 2007

Setting Up Local Fast Deployment Environment with SFTP

Sun, 10 Sep 2017 00:00:00 +0000

Preparation

First, we need to ensure that sshd, our SSH service, is running. The following example uses CentOS to demonstrate the setup process.

Required packages: openssh-clients openssh-server

Run the following command to verify that our SSH service is in active status:

$ systemctl status sshd

Getting Started

Create a new user (add sudo for non-root users):

$ adduser test

$ passwd test

Set password, update user token information, generate SSH keys locally and place the public key on the server to ensure SSH login from local to remote host is possible.

Clarify folder permissions for the new user:

$ chown root:root /../somepath/sftp
$ chmod 755 /../somepath/sftp

Create an uploads folder under the sftp directory where all uploaded files will be stored, and set the owner to the test user:

$ chown test:test /../somepath/sftp/uploads

Set PasswordAuthentication to no and save to ensure SSH connections cannot be established with passwords, then restart the SSH service:

$ vi /etc/ssh/sshd_config
$ ...
$ systemctl restart sshd

Now try logging in locally using sftp:

sftp -i ~/.ssh/key test@ip_address

If you enter sftp successfully, it’s working!

Configuring Fast Deployment Environment with VSCode

We’re using the ftp-sync plugin here.

After installation, press F1:

Select Ftp-sync: init

You can find the ftp-sync.json file under the .vscode folder:

{
    "remotePath": "/../sftp/uploads/project",
    "host": "",	// IP address
    "username": "test",
    "password": null,	// Leave empty as we use the more secure sftp protocol for file transfer
    "port": 22,	// SSH port
    "secure": true,
    "protocol": "sftp",
    "uploadOnSave": false,
    "passive": false,
    "debug": false,
    "privateKeyPath": "/Users/user/.ssh/key",  // Local key path
    "passphrase": null,
    "ignore": [
        "\\.vscode",
        "\\.git",
        "\\.DS_Store",
        "\\node_modules",
        "\\test",
        "\\.travis.yml"		// Folders or files to ignore during upload
    ],
    "generatedFiles": {
        "uploadOnSave": false,
        "extensionsToInclude": [],
        "path": ""
    }
}

Now enjoy fast deployment to the server:

F1

Select Ftp-sync: Local To Remote

Choose full-sync to execute the upload. Success achieved!

Note: For Node.js projects, you can use pm2 to monitor file changes and automatically restart services, achieving one-click online code updates!

Reference:

https://www.digitalocean.com/community/tutorials/how-to-enable-sftp-without-shell-access-on-centos-7

API Testing with Supertest

Sun, 23 Apr 2017 00:00:00 +0000

In Node.js development, after completing API interfaces, testing is often required. Commonly used testing modules are generally mocha and chai. We should also have corresponding request modules to help us better accomplish assertion tasks. Today’s main topic is supertest.

Those who have used superagent should know that supertest has equally concise and elegant usage. This article is based on Express. Without further ado, let’s dive into practical code examples.

GET Request

const assert = require('chai').assert;
const request = require('supertest');
const should = require('should');
const app = require('../../app');
const _ = require('lodash');
// Setting supertest as request is out of habit

// Assertion
describe("get /", function() {
    // it describes the API functionality and expected return content
    it("should respond with ....", function() {
        // Use supertest to perform GET request and verify its returned status code
        return request(app).get('/')
            .set('Accept', 'application/json')  // set request header
            .expect(200)    // expected status code
            .then(function(res) {
                assert.notEqual(_.findIndex(res.body, {
                    // Use assert assertion library for assertions. If lodash can't find the key-value pair in the outermost layer of JSON (for inner content you can continue accessing based on body)
                    // Using equal or to.include.keys('key') is also acceptable
                    'key' : 'value'
                }), -1);
            })
    })
}

POST Request

// No need to repeat the above, we focus on supertest
return request(app).post('/search')
        .send({ key: 'value' })     // x-www-form-urlencoded
        .field('name', 'Jack')      // form-data
        .attach('avatar', 'test/fixtures/homeboy.jpg')  // post file upload

PUT or DELETE Requests

Similarly

request(app).del('/path')
request(app).put('/path')

Python Syntax Fundamentals

Tue, 11 Apr 2017 00:00:00 +0000

Records fundamental Python knowledge points that differ from other programming languages

Boolean Type

True

False

Division

/ Default division is precise division

// This is the typical operation to get the integer part of quotient

Characters (Similar to C ASCII codes)

ord() Character corresponding to encoding

chr() Encoding to character

'aaa'.encode('utf-8') Convert to corresponding encoding format as bytes

'aaa'.decode('utf-8') Convert from bytes to utf-8 encoding

Formatted Output

r = 18.0555
'%.2f%%' %r
# '18.06%'

Data Type: List (Analogous to Array)

Get number of list elements: len(variable_name)

Accessing element with negative index means counting from the end

Append element to end: vary.append(...)

Insert at specified position: vary.insert(1, 'Jack')

Remove last element: vary.pop()

Remove specified position: vary.pop(2)

Elements can have different data types

Data Type: Tuple

Compared to list, arrays defined this way have elements that are not directly mutable, making data more secure

Indirect change principle: tuple keeps the element reference unchanged, but the content being referenced may be mutable, for example when the element is a list

t = ('A', 'B', 'C')

If Statement

: Replaces curly brace blocks

elif Equivalent to else if in other languages

Loops

for-in loop for x in ...:

continue Break current iteration and proceed to next iteration

break Exit the loop body

Dict Key-Value Pairs

d = {'A':11,'B':22,'C':33}
d['B']
# 22

When key is not found, an error occurs, so check key in d return value before searching

d.get('B', -1) Second parameter specifies the value to return when not found

key in dict must be immutable

Delete: d.pop(key)

Set Key Collection

Initialization requires passing a list, input elements cannot have duplicates

s = set([5,5,6,7,7,6,7])
# {5, 6, 7}

Add: add(key)

Remove: remove(key)

String Object

str.replace('a', 'A') Replace a with A but the method returns a new string, the original str is immutable!

Basic Functions

abs()

max() Can accept any number of parameters and only returns the maximum

a = abs Assign abs function reference to a, calling a() has the same effect

Type Conversion

int(variable_to_convert)

Others similar

Function Definition

def Replaces function

Empty Indented Block

pass

Function Return Values

Can actually have multiple!! My worldview…

def goBack(x,y)
	return x,y

a,b = goBack(1,2)

Reason: Returns tuple

Type Checking

def goBack(x,y)
	if isinstance(x, (int, float)):
		return x,y
# Allows int and float types

Parameters

def power(x, n=2):
# n defaults to 2

Default parameters must point to immutable objects

Parameters can be list or tuple, using variable parameters

Variable parameters def calc(*numbers): Multiple parameters automatically assembled into tuple

Keyword parameters def person(name, age, **kw): Convenient for extending function parameters

The order of parameter definition must be: required parameters, default parameters, variable parameters, named keyword parameters, and keyword parameters

Tail Recursion Optimization for Recursive Functions

Refers to calling itself when returning from a function, and the return statement cannot contain expressions

Note: Standard interpreter does not support this usage

Common Concepts in Machine Learning

Sat, 08 Apr 2017 00:00:00 +0000

Features (Attributes)

Usually the columns of the training sample set, which can be viewed as column names for each column. For example, to distinguish bird species, features such as weight and back color can be used for differentiation.

Feature Instance

The data within a specific feature column

Types of Features

Numerical
Binary (similar to Boolean type)
Categorical (color values)

Training Set

A collection of data samples used to train machine learning algorithms

Training Sample

Each row of the sample set is a training sample

Target Variable

The prediction result of the machine learning algorithm

Classification:

In classification algorithms, typically nominal, the target variable is called class
In regression algorithms, typically continuous

Before training the sample set, the value of the target variable must be determined. For example, based on features like bird height, weight, and color, the specific bird species can be determined. The species is the target variable, and the species name is the specific value of the target variable.

Knowledge Representation

Refers to the work of presenting machine classification results

Forms include: rule sets probability distributions instances in training sample sets

Why?

So that machine learning algorithms can discover relationships between features and target variables

The main task of machine learning is classification

Dividing instance data into appropriate categories

Basic classification process:

Obtain all feature information
Algorithm training (learning how to classify)
Testing the effectiveness of machine learning algorithms

How to test algorithm effectiveness?

To test effectiveness, two separate sample sets are typically used: training data and test data

Another important task in machine learning: Regression

Mainly used for predicting numerical data

Program starts running:

Training sample set provides target variable => Input to algorithm => Training completed => Input test data (without target variable) => Compare test results with actual target variable differences (regression fitting) => Obtain actual accuracy of the algorithm

Supervised Learning vs Unsupervised Learning

Supervised learning means the algorithm knows what to predict, i.e., has clear objectives

Regression and classification both belong to supervised learning

Compared to supervised learning, unsupervised learning has no target values

Unsupervised Learning

Purpose:

Reduce the dimensionality of data features

Clustering (Analysis)

Divide similar objects into different groups or more subsets through static classification methods. This is equivalent to dividing one training set into multiple training sets, where the data features in each new training set are similar.

Density Estimation

Refers to the process of finding statistical descriptions of data. It can be understood as obtaining a frequency distribution histogram that describes the data, reducing feature dimensions to make it more intuitive.

Brief Discussion on Algorithm Selection

Goal-oriented: Choose supervised or unsupervised learning algorithms based on the task to be accomplished
Data source-oriented: Analyze or collect what kind of data is needed

General Steps for Developing Machine Learning Applications

Collect data
Prepare input data
Analyze input data (manual)
Train algorithm (machine learning)
Test algorithm
Use algorithm

Problems Caused by Multiple Callbacks in Express

Thu, 23 Mar 2017 00:00:00 +0000

Error Result

Error: Can’t render headers after they are sent to the client.

Error: Response headers cannot be properly rendered when they have already been sent back to the client!

Occurrence Scenario

When using mongoose models for fuzzy queries in Express environment, ORM requires at least one and up to two queries, all of which use Promise-based asynchronous operations.

Problem Troubleshooting

Conflict occurred in response header settings
The error is from res.send(docs)
When one of the Promise queries was removed during debugging, the error disappeared
The issue lies with Promises

Problem Analysis

Why do two consecutive Promise callbacks ultimately cause response header rendering conflicts?

The rendering conflict appears at the top of the error, related to the order of error throwing, which is associated with the built-in res object in Express. The res inherits from node.js native http.ServerResponse class. Before res calls res.writeHead(statusCode) to write the status code of response headers, we can freely write header information. In a res.send(docs), the following steps should be included, some belong to Express itself, others belong to node’s native http module.

Running in the following order:

res.writeContinue()

res.statusCode = 404

res.setHeader(name, value)

res.getHeader(name)

res.removeHeader(name)

res.header(key[, val]) (Express only)

res.charset = ‘utf-8’ (Express only)

res.contentType(type) (Express only)

res.send([body]) (Express only)

Finally, our query data is returned to the client in the form of body in response content. When performing send, node’s built-in functions will run first. Of course, running the first Promise has no problem, but the issue lies in: send itself cannot interrupt the current executing task. As long as there’s no return, the code will continue to run. This precisely confirms the characteristics of node’s asynchronous non-blocking IO. Even if the first Promise executes, it won’t block the continued execution of the second Promise. If the first Promise’s res.send has already executed, when the second Promise is executed and reaches the third step - setting response headers - the original response header information still exists, violating the rule in node that response headers cannot be set repeatedly. Therefore, node throws an error first, making everything seem natural.

All in all, what’s the solution?

Avoid using multiple send operations; this error occurs when setting response headers multiple times
Use asynchronous Promises carefully; consider Promise nesting (recommended)
Remember to add return before send so that subsequent code won’t run, but this method is difficult to control. When asynchronous operations are written together, who knows which one will finish first?

Lexicographic Sorting Method for Strings with Mixed Numbers, Uppercase and Lowercase Letters

Fri, 04 Nov 2016 00:00:00 +0000

Core Steps Overview

Filtering

Filter out numbers and other special symbols, keeping only letters

The replaceAll method built into the String class and regular expressions will be used for filtering

Splitting

Split the string into single characters

The substring method built into the String class will be used here

Sorting

Sort the single-character String array obtained from splitting in a case-insensitive manner

The sort method in the Arrays class will be used here

Key Code Demonstration

Preface

Since the number of input lines and string content are arbitrary, each line’s string is directly added to Java’s ArrayList dynamic array, then transferred to a String array newArray initialized with length, thus avoiding the risk of forced type conversion from Object type to String[].

String Filtering Demonstration

for (int i=0;i <elementIndex;i++)
newArray[i] = newArray[i].replaceAll("\\d+","");
//This demonstrates removing numbers

Extension:

.replaceAll("escape character + regular expression", "replacement character");

String Splitting Demonstration

for(int k=0;k<newArray[number].length();k++)
sortArray[k] = newArray[number].substring(k,k+1);
//Use a for loop to cut one character at a time and add it to sortArray

Extension:

Common String Class Methods

Method Purpose	Definition
Extract substring from start to end index	substring(int beginIndex,int endIndex)
Use a character as delimiter	split(String regex) return String[]
Convert to uppercase letters	toUpperCase()
Convert to lowercase letters	toLowerCase()
Remove leading and trailing spaces	trim() return a string
Compare strings	boolean equals(Object anObject)

Single Character String Array Sorting Demonstration

Arrays.sort(sortArray,String.CASE_INSENSITIVE_ORDER);
//Sort case-insensitively

Extension:

Common Arrays Class Methods

Method Purpose	Definition
Sort array	sort()
Find value in array	binarySearch(object[ ], object key) *First use sort() then search
Copy array	int[] copyOf(int[] original,int newLength)
Replace values in array	fill(int[] a,int val)
To be continued….

Complete Code Demo

/**
 * Created by Tecker on 2016/6/21.
 */
import java.util.*;

public class UsingArrayList {
    public static void main(String[] args){
        int elementIndex = 1;
        Scanner inputElement = new Scanner(System.in);
        ArrayList elementlist = new ArrayList();
        boolean inputOver = false;
        System.out.printf("Please enter N lines of N strings\n");
        System.out.printf("End when inputOver is entered\n");

        do {
            System.out.printf("Enter the " + elementIndex + "th string\n");
            System.out.printf(">>");
            String input = inputElement.nextLine();
            if (input.equals("inputOver")){
                inputOver = true;
                System.out.println("Input completed!");
            }
            else {
                elementlist.add(input);
                elementIndex++;
            }
        }while (inputOver == false);      //Use do-while statement with if judgment to control input of N lines of mixed case strings with numbers
        elementIndex = elementIndex - 1;
        System.out.printf("Before sorting:\n");
        String[] newArray = new String[elementIndex];
        for (int k=0;k <elementIndex;k++)
            System.out.printf(">> " + elementlist.get(k) + "\n");
        for (int index=0;index <elementIndex;index++)
            newArray[index] = elementlist.get(index).toString();
        for (int i=0;i <elementIndex;i++)
            newArray[i] = newArray[i].replaceAll("\\d+","");
        for (String element : newArray)
            System.out.printf(element +"\n");
        System.out.println("After sorting\n");
        for (int listIndex=0;listIndex <newArray.length;listIndex++){
            sortArrays(listIndex,newArray);
        }


    }
    public static void sortArrays(int number,String[] newArray){
        String[] sortArray = new String[newArray[number].length()];
        for(int k=0;k<newArray[number].length();k++){
            sortArray[k] = newArray[number].substring(k,k+1);
        }
        Arrays.sort(sortArray,String.CASE_INSENSITIVE_ORDER);
        String finalString = ">>";
        for (int j=0;j<sortArray.length;j++){
            finalString = finalString.concat(sortArray[j]);
        }
        System.out.println(finalString);
    }

}

FloatShell – Tecker's Universe

Use These 4 Principles to Improve AI Coding

Why Your AI Keeps Repeating the Same Mistakes

A Real Example

Memory Systems Do Not Solve Team Collaboration

Stop Overusing /init

Karpathy’s 4 Rules

Why 4 Rules Can Be Better Than 20

From 4 Rules to 12: When to Expand

Project Rules Should Not Become an Encyclopedia

The Root CLAUDE.md Is More Like Onboarding

When to Update It

Start Now

Letter to Spiritual Shareholders 2026

Preface

Change in the Internet World

AI Agent Production Cycle Mechanism

Consumption-Side Change

The Power of Words

Looking Forward

Docker Architecture Deep Dive

Namespaces

Network

libnetwork

Mount Points

CGroups

UnionFS

Docker Containers and Container Cloud Study Notes

Namespace

UTS namespace

IPC namespace

PID namespace

Regular Expression Quick Start Guide

Same Matching Purpose but Different Implementations

Multiple Matching

Case Sensitivity

Matching Any Character

Matching a Set of Characters

Metacharacters

Matching Specific Character Classes

Matching Whitespace Characters

Matching Multiple Characters

Matching 0 or 1 Character

Limiting Repetition Count

Greedy and Lazy Metacharacters

Position Matching

Boundary Anchors

Multiline Matching Mode

Subexpressions

Nested Subexpressions

Backreferences

Using Backreferences for Replacement

Lookahead and Lookbehind

Embedded Conditions

Backreference Conditions

Lookahead/Lookbehind Conditions

Principles and Techniques for Prompt Writing in Context Engineering

Core Principles for Prompt Writing: Clarity and Minimalism

Extremely Clear and Direct Language

Pursuing Minimal Information Collections

Key Technique: Finding “The Right Altitude”

Failure Mode One: Too Rigid (Hard-coded Complex Logic)

Failure Mode Two: Too Vague (Fuzzy Guidance)

Prompts at “The Right Altitude”

Structured Prompts: Leveraging Tags to Enhance Agent Readability

Carefully Curated Few-shot Prompting

Best Practices Guide for Writing Efficient Tools (Essential Edition)

Introduction

Core Methodology: Build, Evaluate, and Collaborate Optimization

Key Principles for Writing Efficient Tools

Outlook

Meta-Prompting Techniques for Enhanced AI Output Quality

Background

What are Meta-Prompts?

Common Meta-Prompt Templates

Summary

Best Practices for Wrapping C++ Libraries with CGO

Background

Use CGO Only When Necessary

C ABI Interface Encapsulation Follows Minimalism Principle

Stop Overusing `/init`