Skip to content

Create Nested(...) columns as Array(Tuple(...)) - #575

Open
Hakaou wants to merge 1 commit into
ClickHouse:masterfrom
Hakaou:nested-as-array-of-tuple
Open

Hakaou wants to merge 1 commit into
ClickHouse:masterfrom
Hakaou:nested-as-array-of-tuple

Conversation

@Hakaou

@Hakaou Hakaou commented Oct 17, 2026 •

Copy link
Copy Markdown

Closes #571.

Summary

TypeParser had no entry for Nested, so the name became a Void terminal. At the root ValidateAST rejects it ("unsupported column type"), but inside a container nothing checks it: Array(Nested(key String, value String)) was created as Array(Void), ColumnNothing reads one byte per row and the block stream desynchronises.

Nested is now its own AST node, next to Map and SimpleAggregateFunction, and the column factory builds it in the Tuple case, wrapped in an Array. That is how the server defines it: DataTypeNested.cpp builds DataTypeArray(DataTypeTuple(types, names)) under a custom name. Sharing the Tuple case keeps the AST read by reference, so a deeply nested type is not copied at every level, the pattern #588 removes from LowCardinality.

Behavior changes

  • Array(Nested(...)) and Map(K, Nested(...)) columns decode, with their field names kept on the Tuple.
  • A top-level Nested column, which the server sends with flatten_nested = 0, decodes instead of failing with "unsupported column type".
  • The created column reports the structural type name Array(Tuple(key String, value UInt8)) rather than Nested(...). The server accepts that name on INSERT: a block selected from such a column goes back in as is (ArrayOfNested_ReinsertSelectedRows), and a block built with unnamed tuples goes in too (checked against 25.12, 26.3 and 26.11). A dedicated Nested column type would add public API for no behavioural gain, so I did not go that way. Happy to discuss.
  • TypeAst::Meta gains the Nested value.

Tests

  • type_parser_ut: the Nested node with its named elements, and a field itself named Nested.
  • CreateColumnByType_ut: Nested, Array(Nested), Map(String, Nested) and Nested(a Nested(...)) give the expected column types. DeepNestedDoesNotCopyAst builds 200 levels of Nested over 8 000 fields and checks that peak RSS grows by less than 16 MiB. When the node is copied at each level it grows by about 180 MiB on macOS arm64 and 215 MiB on Linux aarch64. The measure runs in a child process (EXPECT_EXIT, hence the DeathTest suite), so a peak that an earlier test reached and released does not count. It reads getrusage, so it runs on Linux and macOS and skips on Windows.
  • client_ut, both fixtures (no compression and LZ4): select from a server-typed Array(Nested(...)) column with empty inner and outer arrays, insert a selected block back as is and read it back with toString, and a top-level Nested column with flatten_nested = 0.

Each new test fails when the Nested entry in GetTypeMeta is removed. Before this change, on master fc5d98f:

Case Before After
CreateColumnByType("Array(Nested(key String, value UInt8))") Array(Void) Array(Array(Tuple(key String, value UInt8)))
Select on that column, no compression Array(Void) in the callback, then an empty server exception 3 rows, values match
Select on that column, LZ4 some data was not read 3 rows, values match
Select on a Nested column, flatten_nested = 0 unsupported column type: Nested(key String, value UInt8) 2 rows, values match
Select, then insert the block back fails at the Select both rows read back identically

Full suite green locally (macOS arm64, Apple clang 21) against ClickHouse 26.11.1 master build, except the Remote* tests, which need external TLS endpoints, and Socketcase.connecttimeout, not run. The new client tests also pass against 25.12 and 26.3. Under AddressSanitizer and UBSan, the parser and factory suites pass on this branch, and the client and roundtrip suites with #590 on top, UBSan reporting only the InsertData reads noted below. The branch also builds without warnings with GCC 14 on Linux aarch64, where the server-free suite passes.

Out of scope

@CLAassistant

CLAassistant commented Oct 17, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

return std::make_shared<ColumnTuple>(columns, std::move(names));
auto tuple = any_named ? std::make_shared<ColumnTuple>(columns, std::move(names))
: std::make_shared<ColumnTuple>(columns);
if (ast.meta == TypeAst::Nested) {

@Hakaou Hakaou Oct 19, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nested takes the Tuple path and is wrapped here, so its elements are read from the AST by reference. An earlier version of this PR copied the node into a Tuple first, which costs the subtree at every level: about 180 MiB (macOS arm64) to 215 MiB (Linux aarch64) of peak RSS for 200 levels over 8 000 fields, the pattern #588 removes from LowCardinality. DeepNestedDoesNotCopyAst holds it.

@Hakaou
Hakaou force-pushed the nested-as-array-of-tuple branch from 10aa714 to b414b18 Compare October 19, 2026 15:50
The type parser had no entry for Nested, so a column typed
Array(Nested(key String, value String)) was created as Array(Void)
and the native block stream fell out of sync: an empty server
exception without compression, "some data was not read" with LZ4.
A top-level Nested column, which the server sends with
flatten_nested = 0, was rejected as unsupported.

The server defines Nested as a custom name over Array(Tuple(...))
(DataTypeNested.cpp). The parser now recognises Nested as its own
node, like Map and SimpleAggregateFunction, and the column factory
builds it in the Tuple case, wrapped in an Array. Sharing that case
keeps the AST read by reference, so a deeply nested type is not
copied at every level.

Closes ClickHouse#571
@Hakaou
Hakaou force-pushed the nested-as-array-of-tuple branch from b414b18 to 71399b4 Compare October 20, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Array(Nested(...)) columns silently decode as Array(Nothing) and desynchronize the native block stream

2 participants