$ impala_cluster $ impala -d oreilly; Starting Impala Shell without Kerberos authentication Welcome to the Impala shell. Press TAB twice to see a list of available commands. Copyright (c) 2012 Cloudera, Inc. All rights reserved. (Shell build version: Impala Shell v2.1.0-cdh5 (e48c2b4) built on Tue Dec 16 19:00:35 PST 2014) [localhost:21000] > desc billion_rows_text; +------+--------+---------+ | name | type | comment | +------+--------+---------+ | x | bigint | | | y | bigint | | | z | bigint | | +------+--------+---------+ [localhost:21000] > select count(*) from billion_rows_text; +------------+ | count(*) | +------------+ | 1000000000 | +------------+ [localhost:21000] > select min(x), min(y), min(z), max(x), max(y), max(z) from billion_rows_text; +--------+--------+--------+----------+---------+--------+ | min(x) | min(y) | min(z) | max(x) | max(y) | max(z) | +--------+--------+--------+----------+---------+--------+ | 398089 | 35496 | 790602 | 36333823 | 7942452 | 790711 | +--------+--------+--------+----------+---------+--------+ [localhost:21000] > profile; Query Runtime Profile: Query (id=1846f10ae6ef341e:3017798f1f6e78a5): Summary: Session ID: 8f4cad0b2b385c70:b41743dae24d22ad Session Type: BEESWAX Start Time: 2015-01-12 15:11:14.621837000 End Time: 2015-01-12 15:11:24.519472000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) User: jrussell Connected User: jrussell Delegated User: Network Address: 127.0.0.1:38627 Default Db: oreilly Sql Statement: select min(x), min(y), min(z), max(x), max(y), max(z) from billion_rows_text Coordinator: a1730.halxg.cloudera.com:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=3.76GB VCores=1 WARNING: The following tables are missing relevant table and/or column statistics. oreilly.billion_rows_text F01:PLAN FRAGMENT [UNPARTITIONED] 03:AGGREGATE [FINALIZE] | output: min:merge(x), min:merge(y), min:merge(z), max:merge(x), max:merge(y), max:merge(z) | hosts=4 per-host-mem=unavailable | tuple-ids=1 row-size=48B cardinality=1 | 02:EXCHANGE [UNPARTITIONED] hosts=4 per-host-mem=unavailable tuple-ids=1 row-size=48B cardinality=1 F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=02, UNPARTITIONED] 01:AGGREGATE | output: min(x), min(y), min(z), max(x), max(y), max(z) | hosts=4 per-host-mem=10.00MB | tuple-ids=1 row-size=48B cardinality=1 | 00:SCAN HDFS [oreilly.billion_rows_text, RANDOM] partitions=1/1 files=113 size=21.75GB table stats: unavailable column stats: unavailable hosts=4 per-host-mem=3.75GB tuple-ids=0 row-size=24B cardinality=unavailable ---------------- Estimated Per-Host Mem: 4037017600 Estimated Per-Host VCores: 1 Tables Missing Stats: oreilly.billion_rows_text Admission result: Admitted immediately Request Pool: root.jrussell ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail -------------------------------------------------------------------------------------------------------------------- 03:AGGREGATE 1 193.586ms 193.586ms 1 1 28.00 KB -1.00 B FINALIZE 02:EXCHANGE 1 61.79us 61.79us 4 1 0 -1.00 B UNPARTITIONED 01:AGGREGATE 4 7s163ms 7s547ms 4 1 116.00 KB 10.00 MB 00:SCAN HDFS 4 1s597ms 1s775ms 1.00B -1 1.43 GB 3.75 GB oreilly.billion_rows_text Query Timeline: 9s900ms - Start execution: 64.467us (64.467us) - Planning finished: 46.226ms (46.161ms) - Submit for admission: 46.791ms (564.917us) - Completed admission: 46.858ms (66.928us) - Ready to start remote fragments: 241.178ms (194.320ms) - Remote fragments started: 437.189ms (196.10ms) - Rows available: 9s855ms (9s418ms) - First row fetched: 9s894ms (38.826ms) - Unregister query: 9s897ms (3.51ms) ImpalaServer: - ClientFetchWaitTimer: 41.143ms - RowMaterializationTimer: 7.773us Execution Profile 1846f10ae6ef341e:3017798f1f6e78a5:(Total: 9s808ms, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 4, last: 0.195184, min: 0.193804, max: 0.195398, mean: 0.194721, stddev: 0.00062514 Per Node Peak Memory Usage: a1733.halxg.cloudera.com:22000(1.31 GB) a1731.halxg.cloudera.com:22000(1.39 GB) a1732.halxg.cloudera.com:22000(1.43 GB) a1730.halxg.cloudera.com:22000(731.47 MB) - FinalizationTimer: 0ns Coordinator Fragment F01:(Total: 9s388ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 28.00 KB, 28.00 KB, 28.00 KB, 36.00 KB ThreadUsage(500.0ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 40.00 KB (40960) - PerHostPeakMemUsage: 0 - PrepareTime: 193.487ms - RowsProduced: 1 - TotalCpuTime: 652.783ms - TotalNetworkReceiveTime: 9s194ms - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 415.303ms, non-child: 415.303ms, % non-child: 100.00%) - CodegenTime: 2.205ms - CompileTime: 16.945ms - LoadTime: 94.353us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 206.665ms - PrepareTime: 190.184ms AGGREGATION_NODE (id=3):(Total: 9s388ms, non-child: 193.586ms, % non-child: 2.06%) ExecOption: Codegen Enabled - BuildTime: 4.55us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 28.00 KB (28672) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 EXCHANGE_NODE (id=2):(Total: 9s194ms, non-child: 61.79us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 53.00 B, 53.00 B, 53.00 B, 159.00 B - BytesReceived: 212.00 B (212) - ConvertRowBatchTime: 10.506us - DeserializeRowBatchTimer: 30.868us - FirstBatchArrivalWaitTime: 7s456ms - PeakMemoryUsage: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F00:(Total: 8s761ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 4.82 GB, max: 5.67 GB, avg: 5.44 GB, stddev: 364.68 MB completion times: min:7s681ms max:9s420ms mean: 8s816ms stddev:670.211ms execution rates: min:608.61 MB/sec max:642.88 MB/sec mean:632.25 MB/sec stddev:13.88 MB/sec num instances: 4 - AverageThreadTokens: 36.31 - PeakMemoryUsage: 1.21 GB (1300425728) - PerHostPeakMemUsage: 1.21 GB (1300428800) - PrepareTime: 193.136ms - RowsProduced: 1 - TotalCpuTime: 5m21s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 280.561us - TotalStorageWaitTime: 4s868ms CodeGen:(Total: 436.753ms, non-child: 436.753ms, % non-child: 100.00%) - CodegenTime: 3.413ms - CompileTime: 34.154ms - LoadTime: 83.47us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 212.323ms - PrepareTime: 188.831ms DataStreamSender (dst_id=2):(Total: 51.970us, non-child: 51.970us, % non-child: 100.00%) - BytesSent: 53.00 B (53) - NetworkThroughput(*): 261.61 KB/sec - OverallThroughput: 995.99 KB/sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 25.250us - ThriftTransmitTime(*): 200.562us - UncompressedRowBatchSize: 64.00 B (64) AGGREGATION_NODE (id=1):(Total: 8s760ms, non-child: 7s163ms, % non-child: 81.77%) - BuildTime: 6s730ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 116.00 KB (118784) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s597ms, non-child: 1s597ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.52 - AverageScannerThreadConcurrency: 36.76 - BytesRead: 5.44 GB (5838667196) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.44 GB (5838661564) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.44 GB (5838661564) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 41 - PeakMemoryUsage: 1.21 GB (1300370432) - PerReadThreadRawHdfsThroughput: 1.11 GB/sec - RemoteScanRanges: 5 - RowsRead: 250.00M (250000000) - RowsReturned: 250.00M (250000000) - RowsReturnedRate: 156.96 M/sec - ScanRangesComplete: 67 - ScannerThreadsInvoluntaryContextSwitches: 13.90K (13897) - ScannerThreadsTotalWallClockTime: 5m17s - DelimiterParseTime: 16s475ms - MaterializeTupleTime(*): 27s767ms - ScannerThreadsSysTime: 1s372ms - ScannerThreadsUserTime: 47s667ms - ScannerThreadsVoluntaryContextSwitches: 269.87K (269870) - TotalRawHdfsReadTime(*): 4s911ms - TotalReadThroughput: 627.02 MB/sec BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns Fragment F00: Instance 1846f10ae6ef341e:3017798f1f6e78aa (host=a1730.halxg.cloudera.com:22000):(Total: 9s323ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB MemoryUsage(500.0ms): 16.00 KB, 579.39 MB, 555.36 MB, 555.39 MB, 533.36 MB, 555.36 MB, 555.42 MB, 547.36 MB, 539.39 MB, 259.36 MB, 659.39 MB, 555.39 MB, 547.43 MB, 547.36 MB, 503.31 MB, 483.23 MB, 459.25 MB, 355.18 MB, 24.07 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 23, 21, 20, 20, 2 - AverageThreadTokens: 21.00 - PeakMemoryUsage: 731.46 MB (766988288) - PerHostPeakMemUsage: 731.47 MB (767000576) - PrepareTime: 192.434ms - RowsProduced: 1 - TotalCpuTime: 3m25s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 254.804us - TotalStorageWaitTime: 1s933ms CodeGen:(Total: 477.63ms, non-child: 477.63ms, % non-child: 100.00%) - CodegenTime: 4.748ms - CompileTime: 53.113ms - LoadTime: 90.276us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 235.163ms - PrepareTime: 187.299ms DataStreamSender (dst_id=2):(Total: 52.327us, non-child: 52.327us, % non-child: 100.00%) - BytesSent: 53.00 B (53) - NetworkThroughput(*): 290.27 KB/sec - OverallThroughput: 989.12 KB/sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 26.147us - ThriftTransmitTime(*): 178.311us - UncompressedRowBatchSize: 64.00 B (64) AGGREGATION_NODE (id=1):(Total: 9s323ms, non-child: 7s547ms, % non-child: 80.95%) ExecOption: Codegen Enabled - BuildTime: 7s083ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 116.00 KB (118784) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s775ms, non-child: 1s775ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB Hdfs Read Thread Concurrency Bucket: 0:77.78% 1:16.67% 2:5.556% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:59 ExecOption: Codegen enabled: 59 out of 59 BytesRead(500.0ms): 0, 746.41 MB, 1002.41 MB, 1.31 GB, 1.60 GB, 1.93 GB, 2.25 GB, 2.53 GB, 2.88 GB, 2.92 GB, 3.59 GB, 3.82 GB, 4.13 GB, 4.48 GB, 4.73 GB, 5.01 GB, 5.33 GB, 5.56 GB, 5.59 GB - AverageHdfsReadThreadConcurrency: 0.28 - AverageScannerThreadConcurrency: 21.11 - BytesRead: 5.60 GB (6012072860) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.60 GB (6012070812) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.60 GB (6012070812) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 731.39 MB (766922752) - PerReadThreadRawHdfsThroughput: 1.16 GB/sec - RemoteScanRanges: 2 - RowsRead: 257.43M (257426663) - RowsReturned: 257.43M (257426663) - RowsReturnedRate: 144.95 M/sec - ScanRangesComplete: 59 - ScannerThreadsInvoluntaryContextSwitches: 19.53K (19532) - ScannerThreadsTotalWallClockTime: 3m17s - DelimiterParseTime: 17s045ms - MaterializeTupleTime(*): 28s601ms - ScannerThreadsSysTime: 1s575ms - ScannerThreadsUserTime: 49s404ms - ScannerThreadsVoluntaryContextSwitches: 284.75K (284750) - TotalRawHdfsReadTime(*): 4s811ms - TotalReadThroughput: 602.69 MB/sec Instance 1846f10ae6ef341e:3017798f1f6e78a7 (host=a1731.halxg.cloudera.com:22000):(Total: 9s085ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB MemoryUsage(500.0ms): 24.00 KB, 1.23 GB, 1.15 GB, 1.09 GB, 1.11 GB, 1.11 GB, 1.11 GB, 1.10 GB, 1.09 GB, 1.08 GB, 1.08 GB, 1.07 GB, 1.06 GB, 1.02 GB, 1012.21 MB, 988.21 MB, 684.12 MB, 340.04 MB ThreadUsage(500.0ms): 24, 48, 48, 48, 48, 48, 48, 48, 48, 47, 47, 47, 47, 45, 44, 43, 41, 39 - AverageThreadTokens: 44.89 - PeakMemoryUsage: 1.39 GB (1493684224) - PerHostPeakMemUsage: 1.39 GB (1493684224) - PrepareTime: 193.798ms - RowsProduced: 1 - TotalCpuTime: 6m39s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 291.156us - TotalStorageWaitTime: 5s140ms BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 422.101ms, non-child: 422.101ms, % non-child: 100.00%) - CodegenTime: 3.13ms - CompileTime: 27.891ms - LoadTime: 74.728us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 202.989ms - PrepareTime: 189.689ms DataStreamSender (dst_id=2):(Total: 52.412us, non-child: 52.412us, % non-child: 100.00%) - BytesSent: 53.00 B (53) - NetworkThroughput(*): 276.49 KB/sec - OverallThroughput: 987.52 KB/sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 24.862us - ThriftTransmitTime(*): 187.197us - UncompressedRowBatchSize: 64.00 B (64) AGGREGATION_NODE (id=1):(Total: 9s085ms, non-child: 7s439ms, % non-child: 81.89%) ExecOption: Codegen Enabled - BuildTime: 7s002ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 116.00 KB (118784) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s645ms, non-child: 1s645ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB Hdfs Read Thread Concurrency Bucket: 0:72.22% 1:16.67% 2:11.11% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:53 ExecOption: Codegen enabled: 53 out of 53 BytesRead(500.0ms): 0, 1.23 GB, 1.48 GB, 1.76 GB, 2.09 GB, 2.36 GB, 2.68 GB, 3.03 GB, 3.35 GB, 3.71 GB, 4.02 GB, 4.34 GB, 4.66 GB, 4.96 GB, 5.27 GB, 5.60 GB, 5.65 GB, 5.66 GB - AverageHdfsReadThreadConcurrency: 0.39 - AverageScannerThreadConcurrency: 43.89 - BytesRead: 5.67 GB (6085373526) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.67 GB (6085367382) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.67 GB (6085367382) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 47 - PeakMemoryUsage: 1.39 GB (1493618688) - PerReadThreadRawHdfsThroughput: 1.07 GB/sec - RemoteScanRanges: 6 - RowsRead: 260.57M (260565882) - RowsReturned: 260.57M (260565882) - RowsReturnedRate: 158.36 M/sec - ScanRangesComplete: 53 - ScannerThreadsInvoluntaryContextSwitches: 12.61K (12607) - ScannerThreadsTotalWallClockTime: 6m34s - DelimiterParseTime: 17s125ms - MaterializeTupleTime(*): 28s951ms - ScannerThreadsSysTime: 1s381ms - ScannerThreadsUserTime: 49s623ms - ScannerThreadsVoluntaryContextSwitches: 278.61K (278610) - TotalRawHdfsReadTime(*): 5s316ms - TotalReadThroughput: 643.94 MB/sec Instance 1846f10ae6ef341e:3017798f1f6e78a9 (host=a1732.halxg.cloudera.com:22000):(Total: 8s995ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB MemoryUsage(500.0ms): 0, 708.40 MB, 1.29 GB, 1.10 GB, 1.11 GB, 1.11 GB, 1.11 GB, 1.10 GB, 1.08 GB, 1.06 GB, 1.06 GB, 1.05 GB, 1.04 GB, 1.02 GB, 1.00 GB, 1012.21 MB, 908.21 MB, 572.21 MB, 155.18 MB ThreadUsage(500.0ms): 1, 48, 48, 48, 48, 48, 48, 48, 48, 46, 46, 46, 46, 45, 44, 43, 43, 43, 20 - AverageThreadTokens: 42.47 - PeakMemoryUsage: 1.43 GB (1535627264) - PerHostPeakMemUsage: 1.43 GB (1535627264) - PrepareTime: 192.916ms - RowsProduced: 1 - TotalCpuTime: 6m38s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 254.614us - TotalStorageWaitTime: 4s414ms BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 424.381ms, non-child: 424.381ms, % non-child: 100.00%) - CodegenTime: 2.913ms - CompileTime: 27.981ms - LoadTime: 91.813us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 205.994ms - PrepareTime: 189.9ms DataStreamSender (dst_id=2):(Total: 51.869us, non-child: 51.869us, % non-child: 100.00%) - BytesSent: 53.00 B (53) - NetworkThroughput(*): 265.86 KB/sec - OverallThroughput: 997.86 KB/sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 25.569us - ThriftTransmitTime(*): 194.679us - UncompressedRowBatchSize: 64.00 B (64) AGGREGATION_NODE (id=1):(Total: 8s994ms, non-child: 7s404ms, % non-child: 82.32%) ExecOption: Codegen Enabled - BuildTime: 6s970ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 116.00 KB (118784) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s590ms, non-child: 1s590ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB Hdfs Read Thread Concurrency Bucket: 0:55.56% 1:33.33% 2:5.556% 3:0% 4:5.556% 5:0% 6:0% File Formats: TEXT/NONE:64 ExecOption: Codegen enabled: 64 out of 64 BytesRead(500.0ms): 671.02 MB, 1.44 GB, 1.56 GB, 1.92 GB, 2.26 GB, 2.58 GB, 2.94 GB, 3.25 GB, 3.55 GB, 3.88 GB, 4.20 GB, 4.46 GB, 4.76 GB, 5.10 GB, 5.43 GB, 5.66 GB, 5.66 GB, 5.66 GB - AverageHdfsReadThreadConcurrency: 0.67 - AverageScannerThreadConcurrency: 43.78 - BytesRead: 5.66 GB (6079010449) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.66 GB (6079009425) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.66 GB (6079009425) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 47 - PeakMemoryUsage: 1.43 GB (1535561728) - PerReadThreadRawHdfsThroughput: 1.14 GB/sec - RemoteScanRanges: 1 - RowsRead: 260.29M (260289143) - RowsReturned: 260.29M (260289143) - RowsReturnedRate: 163.69 M/sec - ScanRangesComplete: 64 - ScannerThreadsInvoluntaryContextSwitches: 12.93K (12934) - ScannerThreadsTotalWallClockTime: 6m33s - DelimiterParseTime: 17s310ms - MaterializeTupleTime(*): 28s889ms - ScannerThreadsSysTime: 1s350ms - ScannerThreadsUserTime: 49s704ms - ScannerThreadsVoluntaryContextSwitches: 278.73K (278725) - TotalRawHdfsReadTime(*): 4s981ms - TotalReadThroughput: 644.15 MB/sec Instance 1846f10ae6ef341e:3017798f1f6e78a8 (host=a1733.halxg.cloudera.com:22000):(Total: 7s641ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB MemoryUsage(500.0ms): 16.00 KB, 640.56 MB, 1.15 GB, 966.13 MB, 948.09 MB, 940.08 MB, 932.04 MB, 916.01 MB, 916.04 MB, 908.00 MB, 896.00 MB, 864.21 MB, 852.04 MB, 827.97 MB, 555.87 MB, 243.70 MB ThreadUsage(500.0ms): 1, 48, 45, 42, 41, 40, 40, 39, 39, 39, 39, 38, 37, 36, 35, 31 - AverageThreadTokens: 36.88 - PeakMemoryUsage: 1.31 GB (1405403136) - PerHostPeakMemUsage: 1.31 GB (1405403136) - PrepareTime: 193.396ms - RowsProduced: 1 - TotalCpuTime: 4m43s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 321.671us - TotalStorageWaitTime: 7s985ms BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 423.466ms, non-child: 423.466ms, % non-child: 100.00%) - CodegenTime: 2.977ms - CompileTime: 27.632ms - LoadTime: 75.372us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 205.148ms - PrepareTime: 189.326ms DataStreamSender (dst_id=2):(Total: 51.273us, non-child: 51.273us, % non-child: 100.00%) - BytesSent: 53.00 B (53) - NetworkThroughput(*): 213.82 KB/sec - OverallThroughput: 1009.46 KB/sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 24.422us - ThriftTransmitTime(*): 242.62us - UncompressedRowBatchSize: 64.00 B (64) AGGREGATION_NODE (id=1):(Total: 7s640ms, non-child: 6s262ms, % non-child: 81.96%) ExecOption: Codegen Enabled - BuildTime: 5s866ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 116.00 KB (118784) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s378ms, non-child: 1s378ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB Hdfs Read Thread Concurrency Bucket: 0:66.67% 1:6.667% 2:20% 3:0% 4:6.667% 5:0% 6:0% File Formats: TEXT/NONE:94 ExecOption: Codegen enabled: 94 out of 94 BytesRead(500.0ms): 0, 634.47 MB, 1.36 GB, 1.50 GB, 1.81 GB, 2.18 GB, 2.51 GB, 2.84 GB, 3.15 GB, 3.48 GB, 3.78 GB, 4.08 GB, 4.40 GB, 4.74 GB, 4.82 GB, 4.82 GB - AverageHdfsReadThreadConcurrency: 0.73 - AverageScannerThreadConcurrency: 38.27 - BytesRead: 4.82 GB (5178211951) - BytesReadDataNodeCache: 0 - BytesReadLocal: 4.82 GB (5178198639) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 4.82 GB (5178198639) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 47 - PeakMemoryUsage: 1.31 GB (1405378560) - PerReadThreadRawHdfsThroughput: 1.06 GB/sec - RemoteScanRanges: 13 - RowsRead: 221.72M (221718312) - RowsReturned: 221.72M (221718312) - RowsReturnedRate: 160.86 M/sec - ScanRangesComplete: 94 - ScannerThreadsInvoluntaryContextSwitches: 10.52K (10515) - ScannerThreadsTotalWallClockTime: 4m43s - DelimiterParseTime: 14s421ms - MaterializeTupleTime(*): 24s628ms - ScannerThreadsSysTime: 1s181ms - ScannerThreadsUserTime: 41s939ms - ScannerThreadsVoluntaryContextSwitches: 237.40K (237398) - TotalRawHdfsReadTime(*): 4s536ms - TotalReadThroughput: 617.29 MB/sec [localhost:21000] > select max_int(); +------------+ | max_int() | +------------+ | 2147483647 | +------------+ [localhost:21000] > select max_smallint(); +----------------+ | max_smallint() | +----------------+ | 32767 | +----------------+ [localhost:21000] > alter table billion_rows_text change x x int; [localhost:21000] > alter table billion_rows_text change y y int; [localhost:21000] > alter table billion_rows_text change z z int; [localhost:21000] > select min(x), min(y), min(z), max(x), max(y), max(z) from billion_rows_text; +--------+--------+--------+----------+---------+--------+ | min(x) | min(y) | min(z) | max(x) | max(y) | max(z) | +--------+--------+--------+----------+---------+--------+ | 398089 | 35496 | 790602 | 36333823 | 7942452 | 790711 | +--------+--------+--------+----------+---------+--------+ [localhost:21000] > profile; Query Runtime Profile: Query (id=ff44aa53812682fd:c1abe44cabf0fe9a): Summary: Session ID: 8f4cad0b2b385c70:b41743dae24d22ad Session Type: BEESWAX Start Time: 2015-01-12 15:13:25.181582000 End Time: 2015-01-12 15:13:34.575604000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) User: jrussell Connected User: jrussell Delegated User: Network Address: 127.0.0.1:38627 Default Db: oreilly Sql Statement: select min(x), min(y), min(z), max(x), max(y), max(z) from billion_rows_text Coordinator: a1730.halxg.cloudera.com:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=3.76GB VCores=1 WARNING: The following tables are missing relevant table and/or column statistics. oreilly.billion_rows_text F01:PLAN FRAGMENT [UNPARTITIONED] 03:AGGREGATE [FINALIZE] | output: min:merge(x), min:merge(y), min:merge(z), max:merge(x), max:merge(y), max:merge(z) | hosts=4 per-host-mem=unavailable | tuple-ids=1 row-size=24B cardinality=1 | 02:EXCHANGE [UNPARTITIONED] hosts=4 per-host-mem=unavailable tuple-ids=1 row-size=24B cardinality=1 F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=02, UNPARTITIONED] 01:AGGREGATE | output: min(x), min(y), min(z), max(x), max(y), max(z) | hosts=4 per-host-mem=10.00MB | tuple-ids=1 row-size=24B cardinality=1 | 00:SCAN HDFS [oreilly.billion_rows_text, RANDOM] partitions=1/1 files=113 size=21.75GB table stats: unavailable column stats: unavailable hosts=4 per-host-mem=3.75GB tuple-ids=0 row-size=12B cardinality=unavailable ---------------- Estimated Per-Host Mem: 4037017600 Estimated Per-Host VCores: 1 Tables Missing Stats: oreilly.billion_rows_text Admission result: Admitted immediately Request Pool: root.jrussell ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail ------------------------------------------------------------------------------------------------------------------- 03:AGGREGATE 1 194.137ms 194.137ms 1 1 28.00 KB -1.00 B FINALIZE 02:EXCHANGE 1 189.383us 189.383us 4 1 0 -1.00 B UNPARTITIONED 01:AGGREGATE 4 6s739ms 7s129ms 4 1 68.00 KB 10.00 MB 00:SCAN HDFS 4 1s603ms 1s735ms 1.00B -1 1.44 GB 3.75 GB oreilly.billion_rows_text Query Timeline: 9s396ms - Start execution: 64.230us (64.230us) - Planning finished: 42.860ms (42.796ms) - Submit for admission: 43.439ms (578.733us) - Completed admission: 43.511ms (72.457us) - Ready to start remote fragments: 238.389ms (194.878ms) - Remote fragments started: 435.543ms (197.153ms) - Rows available: 9s350ms (8s915ms) - First row fetched: 9s391ms (40.329ms) - Unregister query: 9s394ms (2.699ms) ImpalaServer: - ClientFetchWaitTimer: 42.299ms - RowMaterializationTimer: 7.675us Execution Profile ff44aa53812682fd:c1abe44cabf0fe9a:(Total: 9s307ms, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 4, last: 0.196293, min: 0.193179, max: 0.196293, mean: 0.194724, stddev: 0.00110408 Per Node Peak Memory Usage: a1733.halxg.cloudera.com:22000(1.27 GB) a1731.halxg.cloudera.com:22000(1.44 GB) a1732.halxg.cloudera.com:22000(1.41 GB) a1730.halxg.cloudera.com:22000(707.13 MB) - FinalizationTimer: 0ns Coordinator Fragment F01:(Total: 8s904ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 28.00 KB, 28.00 KB, 36.00 KB ThreadUsage(500.0ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 40.00 KB (40960) - PerHostPeakMemUsage: 0 - PrepareTime: 194.46ms - RowsProduced: 1 - TotalCpuTime: 636.874ms - TotalNetworkReceiveTime: 8s710ms - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 396.915ms, non-child: 396.915ms, % non-child: 100.00%) - CodegenTime: 2.242ms - CompileTime: 9.435ms - LoadTime: 104.270us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 195.21ms - PrepareTime: 190.997ms AGGREGATION_NODE (id=3):(Total: 8s904ms, non-child: 194.137ms, % non-child: 2.18%) ExecOption: Codegen Enabled - BuildTime: 3.353us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 28.00 KB (28672) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 EXCHANGE_NODE (id=2):(Total: 8s710ms, non-child: 189.383us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 36.00 B, 36.00 B, 108.00 B - BytesReceived: 144.00 B (144) - ConvertRowBatchTime: 8.648us - DeserializeRowBatchTimer: 23.112us - FirstBatchArrivalWaitTime: 7s124ms - PeakMemoryUsage: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F00:(Total: 8s343ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 4.82 GB, max: 5.67 GB, avg: 5.44 GB, stddev: 364.68 MB completion times: min:7s332ms max:8s917ms mean: 8s386ms stddev:619.987ms execution rates: min:642.96 MB/sec max:676.04 MB/sec mean:664.51 MB/sec stddev:13.03 MB/sec num instances: 4 - AverageThreadTokens: 36.34 - PeakMemoryUsage: 1.20 GB (1290794824) - PerHostPeakMemUsage: 1.20 GB (1290797896) - PrepareTime: 193.101ms - RowsProduced: 1 - TotalCpuTime: 5m6s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 345.676us - TotalStorageWaitTime: 3s761ms BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 424.936ms, non-child: 424.936ms, % non-child: 100.00%) - CodegenTime: 2.995ms - CompileTime: 27.698ms - LoadTime: 82.174us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 206.778ms - PrepareTime: 189.71ms DataStreamSender (dst_id=2):(Total: 38.316us, non-child: 38.316us, % non-child: 100.00%) - BytesSent: 36.00 B (36) - NetworkThroughput(*): 140.69 KB/sec - OverallThroughput: 920.15 KB/sec - PeakMemoryUsage: 4.57 KB (4680) - SerializeBatchTime: 12.351us - ThriftTransmitTime(*): 259.673us - UncompressedRowBatchSize: 36.00 B (36) AGGREGATION_NODE (id=1):(Total: 8s343ms, non-child: 6s739ms, % non-child: 80.78%) - BuildTime: 6s301ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 68.00 KB (69632) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s603ms, non-child: 1s603ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.55 - AverageScannerThreadConcurrency: 37.57 - BytesRead: 5.44 GB (5838667196) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.44 GB (5838661564) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.44 GB (5838661564) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 41 - PeakMemoryUsage: 1.20 GB (1290757376) - PerReadThreadRawHdfsThroughput: 1.35 GB/sec - RemoteScanRanges: 11 - RowsRead: 250.00M (250000000) - RowsReturned: 250.00M (250000000) - RowsReturnedRate: 156.25 M/sec - ScanRangesComplete: 67 - ScannerThreadsInvoluntaryContextSwitches: 13.40K (13405) - ScannerThreadsTotalWallClockTime: 5m2s - DelimiterParseTime: 16s810ms - MaterializeTupleTime(*): 27s734ms - ScannerThreadsSysTime: 1s384ms - ScannerThreadsUserTime: 47s670ms - ScannerThreadsVoluntaryContextSwitches: 270.98K (270984) - TotalRawHdfsReadTime(*): 4s034ms - TotalReadThroughput: 676.37 MB/sec Fragment F00: Instance ff44aa53812682fd:c1abe44cabf0fe9f (host=a1730.halxg.cloudera.com:22000):(Total: 8s865ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB MemoryUsage(500.0ms): 16.57 KB, 562.11 MB, 554.08 MB, 548.08 MB, 540.09 MB, 554.11 MB, 554.11 MB, 554.04 MB, 394.06 MB, 546.08 MB, 570.06 MB, 554.08 MB, 546.08 MB, 502.03 MB, 478.06 MB, 458.00 MB, 377.97 MB, 24.07 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 22, 22, 20, 20, 2 - AverageThreadTokens: 20.83 - PeakMemoryUsage: 707.11 MB (741462600) - PerHostPeakMemUsage: 707.13 MB (741474888) - PrepareTime: 193.480ms - RowsProduced: 1 - TotalCpuTime: 3m14s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 350.877us - TotalStorageWaitTime: 1s798ms CodeGen:(Total: 434.940ms, non-child: 434.940ms, % non-child: 100.00%) - CodegenTime: 3.60ms - CompileTime: 28.271ms - LoadTime: 80.504us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 215.820ms - PrepareTime: 189.453ms DataStreamSender (dst_id=2):(Total: 37.914us, non-child: 37.914us, % non-child: 100.00%) - BytesSent: 36.00 B (36) - NetworkThroughput(*): 152.51 KB/sec - OverallThroughput: 927.26 KB/sec - PeakMemoryUsage: 4.57 KB (4680) - SerializeBatchTime: 12.107us - ThriftTransmitTime(*): 230.519us - UncompressedRowBatchSize: 36.00 B (36) AGGREGATION_NODE (id=1):(Total: 8s865ms, non-child: 7s129ms, % non-child: 80.43%) ExecOption: Codegen Enabled - BuildTime: 6s668ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 68.00 KB (69632) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s735ms, non-child: 1s735ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB Hdfs Read Thread Concurrency Bucket: 0:58.82% 1:29.41% 2:11.76% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:59 ExecOption: Codegen enabled: 59 out of 59 BytesRead(500.0ms): 0, 747.94 MB, 1.07 GB, 1.40 GB, 1.74 GB, 2.07 GB, 2.42 GB, 2.73 GB, 2.90 GB, 3.37 GB, 3.73 GB, 4.04 GB, 4.36 GB, 4.66 GB, 4.95 GB, 5.28 GB, 5.54 GB, 5.58 GB - AverageHdfsReadThreadConcurrency: 0.53 - AverageScannerThreadConcurrency: 21.00 - BytesRead: 5.60 GB (6012072860) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.60 GB (6012070812) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.60 GB (6012070812) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 707.07 MB (741412864) - PerReadThreadRawHdfsThroughput: 1.39 GB/sec - RemoteScanRanges: 4 - RowsRead: 257.43M (257426663) - RowsReturned: 257.43M (257426663) - RowsReturnedRate: 148.36 M/sec - ScanRangesComplete: 59 - ScannerThreadsInvoluntaryContextSwitches: 18.53K (18526) - ScannerThreadsTotalWallClockTime: 3m6s - DelimiterParseTime: 17s306ms - MaterializeTupleTime(*): 28s509ms - ScannerThreadsSysTime: 1s490ms - ScannerThreadsUserTime: 49s138ms - ScannerThreadsVoluntaryContextSwitches: 284.79K (284790) - TotalRawHdfsReadTime(*): 4s034ms - TotalReadThroughput: 635.21 MB/sec Instance ff44aa53812682fd:c1abe44cabf0fe9e (host=a1732.halxg.cloudera.com:22000):(Total: 8s674ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB MemoryUsage(500.0ms): 0, 702.72 MB, 1.26 GB, 1.10 GB, 1.11 GB, 1.10 GB, 1.10 GB, 1.10 GB, 1.08 GB, 1.07 GB, 1.05 GB, 1.03 GB, 1.03 GB, 1.01 GB, 1022.63 MB, 990.61 MB, 650.60 MB, 306.44 MB ThreadUsage(500.0ms): 1, 48, 48, 48, 48, 48, 48, 48, 48, 47, 46, 45, 45, 45, 44, 44, 43, 37 - AverageThreadTokens: 43.39 - PeakMemoryUsage: 1.41 GB (1517015624) - PerHostPeakMemUsage: 1.41 GB (1517015624) - PrepareTime: 194.570ms - RowsProduced: 1 - TotalCpuTime: 6m21s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 383.647us - TotalStorageWaitTime: 4s301ms BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 421.275ms, non-child: 421.275ms, % non-child: 100.00%) - CodegenTime: 3.25ms - CompileTime: 27.198ms - LoadTime: 100.543us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 202.203ms - PrepareTime: 190.395ms DataStreamSender (dst_id=2):(Total: 38.697us, non-child: 38.697us, % non-child: 100.00%) - BytesSent: 36.00 B (36) - NetworkThroughput(*): 100.32 KB/sec - OverallThroughput: 908.50 KB/sec - PeakMemoryUsage: 4.57 KB (4680) - SerializeBatchTime: 12.759us - ThriftTransmitTime(*): 350.455us - UncompressedRowBatchSize: 36.00 B (36) AGGREGATION_NODE (id=1):(Total: 8s674ms, non-child: 6s984ms, % non-child: 80.52%) ExecOption: Codegen Enabled - BuildTime: 6s527ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 68.00 KB (69632) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s690ms, non-child: 1s690ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB Hdfs Read Thread Concurrency Bucket: 0:64.71% 1:17.65% 2:11.76% 3:0% 4:5.882% 5:0% 6:0% File Formats: TEXT/NONE:64 ExecOption: Codegen enabled: 64 out of 64 BytesRead(500.0ms): 667.32 MB, 1.44 GB, 1.58 GB, 1.94 GB, 2.30 GB, 2.65 GB, 2.98 GB, 3.32 GB, 3.66 GB, 3.99 GB, 4.31 GB, 4.66 GB, 5.00 GB, 5.33 GB, 5.64 GB, 5.65 GB, 5.66 GB - AverageHdfsReadThreadConcurrency: 0.65 - AverageScannerThreadConcurrency: 44.88 - BytesRead: 5.66 GB (6079010449) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.66 GB (6079009425) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.66 GB (6079009425) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 47 - PeakMemoryUsage: 1.41 GB (1516990464) - PerReadThreadRawHdfsThroughput: 1.35 GB/sec - RemoteScanRanges: 2 - RowsRead: 260.29M (260289143) - RowsReturned: 260.29M (260289143) - RowsReturnedRate: 154.00 M/sec - ScanRangesComplete: 64 - ScannerThreadsInvoluntaryContextSwitches: 12.39K (12388) - ScannerThreadsTotalWallClockTime: 6m16s - DelimiterParseTime: 17s796ms - MaterializeTupleTime(*): 28s924ms - ScannerThreadsSysTime: 1s434ms - ScannerThreadsUserTime: 49s955ms - ScannerThreadsVoluntaryContextSwitches: 280.42K (280416) - TotalRawHdfsReadTime(*): 4s206ms - TotalReadThroughput: 682.04 MB/sec Instance ff44aa53812682fd:c1abe44cabf0fe9c (host=a1731.halxg.cloudera.com:22000):(Total: 8s541ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB MemoryUsage(500.0ms): 16.57 KB, 1.41 GB, 1.10 GB, 1.11 GB, 1.10 GB, 1.10 GB, 1.10 GB, 1.09 GB, 1.08 GB, 1.08 GB, 1.07 GB, 1.06 GB, 1.03 GB, 1010.72 MB, 970.60 MB, 706.52 MB, 354.50 MB ThreadUsage(500.0ms): 1, 48, 48, 48, 48, 48, 48, 48, 48, 47, 47, 47, 46, 45, 43, 41, 40 - AverageThreadTokens: 43.59 - PeakMemoryUsage: 1.44 GB (1542214216) - PerHostPeakMemUsage: 1.44 GB (1542214216) - PrepareTime: 191.550ms - RowsProduced: 1 - TotalCpuTime: 6m16s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 280.862us - TotalStorageWaitTime: 3s433ms BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 422.409ms, non-child: 422.409ms, % non-child: 100.00%) - CodegenTime: 2.931ms - CompileTime: 27.932ms - LoadTime: 70.401us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 205.423ms - PrepareTime: 187.739ms DataStreamSender (dst_id=2):(Total: 35.459us, non-child: 35.459us, % non-child: 100.00%) - BytesSent: 36.00 B (36) - NetworkThroughput(*): 169.41 KB/sec - OverallThroughput: 991.46 KB/sec - PeakMemoryUsage: 4.57 KB (4680) - SerializeBatchTime: 10.899us - ThriftTransmitTime(*): 207.519us - UncompressedRowBatchSize: 36.00 B (36) AGGREGATION_NODE (id=1):(Total: 8s541ms, non-child: 6s939ms, % non-child: 81.24%) ExecOption: Codegen Enabled - BuildTime: 6s504ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 68.00 KB (69632) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s602ms, non-child: 1s602ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB Hdfs Read Thread Concurrency Bucket: 0:81.25% 1:12.5% 2:0% 3:6.25% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:53 ExecOption: Codegen enabled: 53 out of 53 BytesRead(500.0ms): 0, 1.44 GB, 1.50 GB, 1.87 GB, 2.22 GB, 2.58 GB, 2.93 GB, 3.26 GB, 3.60 GB, 3.94 GB, 4.25 GB, 4.55 GB, 4.89 GB, 5.19 GB, 5.52 GB, 5.65 GB, 5.67 GB - AverageHdfsReadThreadConcurrency: 0.31 - AverageScannerThreadConcurrency: 45.25 - BytesRead: 5.67 GB (6085373526) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.67 GB (6085367382) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.67 GB (6085367382) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 47 - PeakMemoryUsage: 1.44 GB (1542164480) - PerReadThreadRawHdfsThroughput: 1.34 GB/sec - RemoteScanRanges: 12 - RowsRead: 260.57M (260565882) - RowsReturned: 260.57M (260565882) - RowsReturnedRate: 162.64 M/sec - ScanRangesComplete: 53 - ScannerThreadsInvoluntaryContextSwitches: 12.23K (12233) - ScannerThreadsTotalWallClockTime: 6m10s - DelimiterParseTime: 17s610ms - MaterializeTupleTime(*): 28s842ms - ScannerThreadsSysTime: 1s390ms - ScannerThreadsUserTime: 49s778ms - ScannerThreadsVoluntaryContextSwitches: 280.33K (280327) - TotalRawHdfsReadTime(*): 4s238ms - TotalReadThroughput: 682.76 MB/sec Instance ff44aa53812682fd:c1abe44cabf0fe9d (host=a1733.halxg.cloudera.com:22000):(Total: 7s293ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB MemoryUsage(500.0ms): 0, 924.75 MB, 1.10 GB, 988.60 MB, 970.57 MB, 946.52 MB, 930.53 MB, 914.47 MB, 914.49 MB, 914.49 MB, 902.49 MB, 878.47 MB, 850.56 MB, 738.41 MB, 346.38 MB ThreadUsage(500.0ms): 1, 48, 45, 43, 42, 41, 40, 39, 39, 39, 39, 39, 37, 36, 35 - AverageThreadTokens: 37.53 - PeakMemoryUsage: 1.27 GB (1362486856) - PerHostPeakMemUsage: 1.27 GB (1362486856) - PrepareTime: 192.804ms - RowsProduced: 1 - TotalCpuTime: 4m36s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 367.318us - TotalStorageWaitTime: 5s514ms BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 0 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 421.123ms, non-child: 421.123ms, % non-child: 100.00%) - CodegenTime: 2.965ms - CompileTime: 27.390ms - LoadTime: 77.249us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 203.667ms - PrepareTime: 188.698ms DataStreamSender (dst_id=2):(Total: 41.197us, non-child: 41.197us, % non-child: 100.00%) - BytesSent: 36.00 B (36) - NetworkThroughput(*): 140.51 KB/sec - OverallThroughput: 853.37 KB/sec - PeakMemoryUsage: 4.57 KB (4680) - SerializeBatchTime: 13.639us - ThriftTransmitTime(*): 250.202us - UncompressedRowBatchSize: 36.00 B (36) AGGREGATION_NODE (id=1):(Total: 7s292ms, non-child: 5s906ms, % non-child: 81.00%) ExecOption: Codegen Enabled - BuildTime: 5s506ms - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 68.00 KB (69632) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 HDFS_SCAN_NODE (id=0):(Total: 1s385ms, non-child: 1s385ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB Hdfs Read Thread Concurrency Bucket: 0:50% 1:35.71% 2:7.143% 3:7.143% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:94 ExecOption: Codegen enabled: 94 out of 94 BytesRead(500.0ms): 881.08 MB, 1.34 GB, 1.55 GB, 1.89 GB, 2.26 GB, 2.56 GB, 2.89 GB, 3.23 GB, 3.56 GB, 3.93 GB, 4.24 GB, 4.60 GB, 4.82 GB, 4.82 GB - AverageHdfsReadThreadConcurrency: 0.71 - AverageScannerThreadConcurrency: 39.14 - BytesRead: 4.82 GB (5178211951) - BytesReadDataNodeCache: 0 - BytesReadLocal: 4.82 GB (5178198639) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 4.82 GB (5178198639) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 47 - PeakMemoryUsage: 1.27 GB (1362461696) - PerReadThreadRawHdfsThroughput: 1.32 GB/sec - RemoteScanRanges: 26 - RowsRead: 221.72M (221718312) - RowsReturned: 221.72M (221718312) - RowsReturnedRate: 160.01 M/sec - ScanRangesComplete: 94 - ScannerThreadsInvoluntaryContextSwitches: 10.47K (10473) - ScannerThreadsTotalWallClockTime: 4m34s - DelimiterParseTime: 14s525ms - MaterializeTupleTime(*): 24s661ms - ScannerThreadsSysTime: 1s220ms - ScannerThreadsUserTime: 41s808ms - ScannerThreadsVoluntaryContextSwitches: 238.41K (238405) - TotalRawHdfsReadTime(*): 3s657ms - TotalReadThroughput: 705.47 MB/sec [localhost:21000] > select count(distinct x) from billion_rows_text; +-------------------+ | count(distinct x) | +-------------------+ | 18 | +-------------------+ [localhost:21000] > profile; Query Runtime Profile: Query (id=ea4d4a38671669c3:4a4814fc2ba66ab7): Summary: Session ID: 8f4cad0b2b385c70:b41743dae24d22ad Session Type: BEESWAX Start Time: 2015-01-12 15:14:01.002146000 End Time: 2015-01-12 15:14:10.898099000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) User: jrussell Connected User: jrussell Delegated User: Network Address: 127.0.0.1:38627 Default Db: oreilly Sql Statement: select count(distinct x) from billion_rows_text Coordinator: a1730.halxg.cloudera.com:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=3.88GB VCores=2 WARNING: The following tables are missing relevant table and/or column statistics. oreilly.billion_rows_text F02:PLAN FRAGMENT [UNPARTITIONED] 06:AGGREGATE [FINALIZE] | output: count:merge(x) | hosts=4 per-host-mem=unavailable | tuple-ids=2 row-size=8B cardinality=1 | 05:EXCHANGE [UNPARTITIONED] hosts=4 per-host-mem=unavailable tuple-ids=2 row-size=8B cardinality=1 F01:PLAN FRAGMENT [HASH(x)] DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=05, UNPARTITIONED] 02:AGGREGATE | output: count(x) | hosts=4 per-host-mem=10.00MB | tuple-ids=2 row-size=8B cardinality=1 | 04:AGGREGATE | group by: x | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=4B cardinality=unavailable | 03:EXCHANGE [HASH(x)] hosts=4 per-host-mem=0B tuple-ids=1 row-size=4B cardinality=unavailable F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=03, HASH(x)] 01:AGGREGATE | group by: x | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=4B cardinality=unavailable | 00:SCAN HDFS [oreilly.billion_rows_text, RANDOM] partitions=1/1 files=113 size=21.75GB table stats: unavailable column stats: unavailable hosts=4 per-host-mem=3.75GB tuple-ids=0 row-size=4B cardinality=unavailable ---------------- Estimated Per-Host Mem: 4160749568 Estimated Per-Host VCores: 2 Tables Missing Stats: oreilly.billion_rows_text Admission result: Admitted immediately Request Pool: root.jrussell ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail -------------------------------------------------------------------------------------------------------------------- 06:AGGREGATE 1 192.137ms 192.137ms 1 1 28.00 KB -1.00 B FINALIZE 05:EXCHANGE 1 58.458us 58.458us 4 1 0 -1.00 B UNPARTITIONED 02:AGGREGATE 4 192.615ms 194.109ms 4 1 12.00 KB 10.00 MB 04:AGGREGATE 4 3.252ms 4.699ms 18 -1 6.27 MB 128.00 MB 03:EXCHANGE 4 69.470us 118.65us 72 -1 0 0 HASH(x) 01:AGGREGATE 4 7s237ms 7s908ms 72 -1 6.29 MB 128.00 MB 00:SCAN HDFS 4 1s302ms 1s465ms 1.00B -1 738.06 MB 3.75 GB oreilly.billion_rows_text Query Timeline: 9s900ms - Start execution: 62.285us (62.285us) - Planning finished: 29.336ms (29.274ms) - Submit for admission: 29.798ms (461.271us) - Completed admission: 29.873ms (75.598us) - Ready to start remote fragments: 222.634ms (192.760ms) - Remote fragments started: 630.172ms (407.538ms) - Rows available: 9s868ms (9s238ms) - First row fetched: 9s885ms (17.490ms) - Unregister query: 9s895ms (10.180ms) ImpalaServer: - ClientFetchWaitTimer: 26.480ms - RowMaterializationTimer: 10.408us Execution Profile ea4d4a38671669c3:4a4814fc2ba66ab7:(Total: 9s838ms, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 8, last: 0.205861, min: 0.194379, max: 0.205861, mean: 0.200437, stddev: 0.00405088 Per Node Peak Memory Usage: a1733.halxg.cloudera.com:22000(678.66 MB) a1731.halxg.cloudera.com:22000(742.66 MB) a1732.halxg.cloudera.com:22000(750.66 MB) a1730.halxg.cloudera.com:22000(494.46 MB) - FinalizationTimer: 0ns Coordinator Fragment F02:(Total: 9s236ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB ThreadUsage(500.0ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 40.00 KB (40960) - PerHostPeakMemUsage: 0 - PrepareTime: 192.63ms - RowsProduced: 1 - TotalCpuTime: 811.51ms - TotalNetworkReceiveTime: 9s044ms - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 384.800ms, non-child: 384.800ms, % non-child: 100.00%) - CodegenTime: 498.436us - CompileTime: 6.592ms - LoadTime: 77.88us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 186.610ms - PrepareTime: 190.231ms AGGREGATION_NODE (id=6):(Total: 9s236ms, non-child: 192.137ms, % non-child: 2.08%) ExecOption: Codegen Enabled - BuildTime: 3.442us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 28.00 KB (28672) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 EXCHANGE_NODE (id=5):(Total: 9s044ms, non-child: 58.458us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 64.00 B (64) - ConvertRowBatchTime: 13.978us - DeserializeRowBatchTimer: 20.551us - FirstBatchArrivalWaitTime: 9s040ms - PeakMemoryUsage: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F01:(Total: 9s390ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:9s445ms max:9s450ms mean: 9s448ms stddev:2.293ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 4 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6609920) - PerHostPeakMemUsage: 666.61 MB (698993664) - PrepareTime: 195.441ms - RowsProduced: 1 - TotalCpuTime: 448.875ms - TotalNetworkReceiveTime: 9s194ms - TotalNetworkSendTime: 418.512us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 443.828ms, non-child: 443.828ms, % non-child: 100.00%) - CodegenTime: 2.516ms - CompileTime: 35.139ms - LoadTime: 78.20us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 216.452ms - PrepareTime: 190.844ms DataStreamSender (dst_id=5):(Total: 38.353us, non-child: 38.353us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 70.14 KB/sec - OverallThroughput: 414.91 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 9.220us - ThriftTransmitTime(*): 248.895us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s390ms, non-child: 192.615ms, % non-child: 2.05%) - BuildTime: 8.347us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s197ms, non-child: 3.252ms, % non-child: 0.04%) - BuildTime: 11.161us - GetNewBlockTime: 183.375us - GetResultsTime: 7.712us - HashBuckets: 4.61K (4608) - LargestPartitionPercent: 6 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6576128) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.196us EXCHANGE_NODE (id=3):(Total: 9s194ms, non-child: 9s194ms, % non-child: 100.00%) - BytesReceived: 222.00 B (222) - ConvertRowBatchTime: 9.87us - DeserializeRowBatchTimer: 27.337us - FirstBatchArrivalWaitTime: 7s426ms - PeakMemoryUsage: 0 - RowsReturned: 18 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F00:(Total: 8s541ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 4.82 GB, max: 5.67 GB, avg: 5.44 GB, stddev: 364.68 MB completion times: min:7s474ms max:9s238ms mean: 8s616ms stddev:707.958ms execution rates: min:624.90 MB/sec max:676.49 MB/sec mean:647.39 MB/sec stddev:21.94 MB/sec num instances: 4 - AverageThreadTokens: 19.82 - PeakMemoryUsage: 660.33 MB (692408320) - PerHostPeakMemUsage: 666.61 MB (698993664) - PrepareTime: 202.110ms - RowsProduced: 18 - TotalCpuTime: 2m51s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 3.236ms - TotalStorageWaitTime: 1s332ms CodeGen:(Total: 471.511ms, non-child: 471.511ms, % non-child: 100.00%) - CodegenTime: 2.700ms - CompileTime: 41.842ms - LoadTime: 75.572us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 230.964ms - PrepareTime: 197.245ms DataStreamSender (dst_id=3):(Total: 38.56us, non-child: 38.56us, % non-child: 100.00%) - BytesSent: 222.00 B (222) - NetworkThroughput(*): 169.34 KB/sec - OverallThroughput: 5.59 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 36.870us - ThriftTransmitTime(*): 1.379ms - UncompressedRowBatchSize: 232.00 B (232) AGGREGATION_NODE (id=1):(Total: 8s540ms, non-child: 7s237ms, % non-child: 84.74%) - BuildTime: 6s815ms - GetNewBlockTime: 152.540us - GetResultsTime: 24.50us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 1.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.82us HDFS_SCAN_NODE (id=0):(Total: 1s302ms, non-child: 1s302ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.40 - AverageScannerThreadConcurrency: 19.99 - BytesRead: 5.44 GB (5838667196) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.44 GB (5838661564) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.44 GB (5838661564) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 21 - PeakMemoryUsage: 654.00 MB (685772800) - PerReadThreadRawHdfsThroughput: 1.57 GB/sec - RemoteScanRanges: 11 - RowsRead: 250.00M (250000000) - RowsReturned: 250.00M (250000000) - RowsReturnedRate: 192.50 M/sec - ScanRangesComplete: 67 - ScannerThreadsInvoluntaryContextSwitches: 9.34K (9339) - ScannerThreadsTotalWallClockTime: 2m43s - DelimiterParseTime: 13s709ms - MaterializeTupleTime(*): 9s580ms - ScannerThreadsSysTime: 1s284ms - ScannerThreadsUserTime: 27s157ms - ScannerThreadsVoluntaryContextSwitches: 269.56K (269564) - TotalRawHdfsReadTime(*): 3s469ms - TotalReadThroughput: 660.64 MB/sec Fragment F01: Instance ea4d4a38671669c3:4a4814fc2ba66abb (host=a1732.halxg.cloudera.com:22000):(Total: 9s401ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 750.66 MB (787128320) - PrepareTime: 194.606ms - RowsProduced: 1 - TotalCpuTime: 437.665ms - TotalNetworkReceiveTime: 9s206ms - TotalNetworkSendTime: 846.973us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 433.39ms, non-child: 433.39ms, % non-child: 100.00%) - CodegenTime: 2.328ms - CompileTime: 30.374ms - LoadTime: 84.271us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 210.822ms - PrepareTime: 190.400ms DataStreamSender (dst_id=5):(Total: 40.817us, non-child: 40.817us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 37.89 KB/sec - OverallThroughput: 382.81 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.869us - ThriftTransmitTime(*): 412.345us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s401ms, non-child: 192.226ms, % non-child: 2.04%) ExecOption: Codegen Enabled - BuildTime: 6.178us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s208ms, non-child: 2.779ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 9.277us - GetNewBlockTime: 95.951us - GetResultsTime: 7.158us - HashBuckets: 3.07K (3072) - LargestPartitionPercent: 8 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 3 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.341us EXCHANGE_NODE (id=3):(Total: 9s206ms, non-child: 51.621us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 40.00 B, 40.00 B, 80.00 B - BytesReceived: 160.00 B (160) - ConvertRowBatchTime: 8.629us - DeserializeRowBatchTimer: 23.905us - FirstBatchArrivalWaitTime: 7s438ms - PeakMemoryUsage: 0 - RowsReturned: 12 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance ea4d4a38671669c3:4a4814fc2ba66aba (host=a1733.halxg.cloudera.com:22000):(Total: 9s399ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6606848) - PerHostPeakMemUsage: 678.66 MB (711630848) - PrepareTime: 193.61ms - RowsProduced: 1 - TotalCpuTime: 438.151ms - TotalNetworkReceiveTime: 9s205ms - TotalNetworkSendTime: 308.711us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 433.609ms, non-child: 433.609ms, % non-child: 100.00%) - CodegenTime: 2.284ms - CompileTime: 30.945ms - LoadTime: 68.485us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 212.254ms - PrepareTime: 189.51ms DataStreamSender (dst_id=5):(Total: 37.685us, non-child: 37.685us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 77.37 KB/sec - OverallThroughput: 414.62 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 8.988us - ThriftTransmitTime(*): 201.953us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s398ms, non-child: 190.790ms, % non-child: 2.03%) ExecOption: Codegen Enabled - BuildTime: 10.642us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s208ms, non-child: 2.795ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 15.68us - GetNewBlockTime: 87.265us - GetResultsTime: 9.716us - HashBuckets: 5.12K (5120) - LargestPartitionPercent: 5 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6569984) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.40us EXCHANGE_NODE (id=3):(Total: 9s205ms, non-child: 61.925us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 61.00 B, 61.00 B, 61.00 B, 122.00 B - BytesReceived: 244.00 B (244) - ConvertRowBatchTime: 8.752us - DeserializeRowBatchTimer: 28.220us - FirstBatchArrivalWaitTime: 7s436ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance ea4d4a38671669c3:4a4814fc2ba66ab9 (host=a1731.halxg.cloudera.com:22000):(Total: 9s396ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 742.66 MB (778739712) - PrepareTime: 195.707ms - RowsProduced: 1 - TotalCpuTime: 441.222ms - TotalNetworkReceiveTime: 9s200ms - TotalNetworkSendTime: 307.624us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 436.738ms, non-child: 436.738ms, % non-child: 100.00%) - CodegenTime: 2.320ms - CompileTime: 30.577ms - LoadTime: 73.855us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 213.192ms - PrepareTime: 191.580ms DataStreamSender (dst_id=5):(Total: 30.720us, non-child: 30.720us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 75.12 KB/sec - OverallThroughput: 508.63 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 4.720us - ThriftTransmitTime(*): 207.989us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s396ms, non-child: 193.334ms, % non-child: 2.06%) ExecOption: Codegen Enabled - BuildTime: 6.649us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s202ms, non-child: 2.735ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 7.447us - GetNewBlockTime: 96.632us - GetResultsTime: 6.118us - HashBuckets: 3.07K (3072) - LargestPartitionPercent: 8 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 3 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.183us EXCHANGE_NODE (id=3):(Total: 9s200ms, non-child: 46.270us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 40.00 B, 40.00 B, 80.00 B, 120.00 B - BytesReceived: 160.00 B (160) - ConvertRowBatchTime: 6.992us - DeserializeRowBatchTimer: 16.774us - FirstBatchArrivalWaitTime: 7s432ms - PeakMemoryUsage: 0 - RowsReturned: 12 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance ea4d4a38671669c3:4a4814fc2ba66abc (host=a1730.halxg.cloudera.com:22000):(Total: 9s364ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 494.46 MB (518475776) - PrepareTime: 198.391ms - RowsProduced: 1 - TotalCpuTime: 478.464ms - TotalNetworkReceiveTime: 9s165ms - TotalNetworkSendTime: 210.743us - TotalStorageWaitTime: 0ns CodeGen:(Total: 471.927ms, non-child: 471.927ms, % non-child: 100.00%) - CodegenTime: 3.130ms - CompileTime: 48.659ms - LoadTime: 85.472us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 229.541ms - PrepareTime: 192.344ms DataStreamSender (dst_id=5):(Total: 44.190us, non-child: 44.190us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 90.16 KB/sec - OverallThroughput: 353.59 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 19.306us - ThriftTransmitTime(*): 173.296us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s364ms, non-child: 194.109ms, % non-child: 2.07%) ExecOption: Codegen Enabled - BuildTime: 9.919us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s170ms, non-child: 4.699ms, % non-child: 0.05%) ExecOption: Codegen Enabled - BuildTime: 12.855us - GetNewBlockTime: 453.652us - GetResultsTime: 7.857us - HashBuckets: 7.17K (7168) - LargestPartitionPercent: 3 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 7 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.220us EXCHANGE_NODE (id=3):(Total: 9s165ms, non-child: 118.65us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 81.00 B, 81.00 B, 162.00 B, 243.00 B - BytesReceived: 324.00 B (324) - ConvertRowBatchTime: 11.977us - DeserializeRowBatchTimer: 40.450us - FirstBatchArrivalWaitTime: 7s398ms - PeakMemoryUsage: 0 - RowsReturned: 28 - RowsReturnedRate: 3.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Fragment F00: Instance ea4d4a38671669c3:4a4814fc2ba66abf (host=a1732.halxg.cloudera.com:22000):(Total: 9s169ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB MemoryUsage(500.0ms): 0, 736.39 MB, 560.39 MB, 560.39 MB, 560.39 MB, 530.39 MB, 568.39 MB, 544.39 MB, 560.36 MB, 416.39 MB, 494.36 MB, 568.39 MB, 568.39 MB, 560.39 MB, 528.39 MB, 536.36 MB, 516.36 MB, 484.33 MB, 264.23 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 23, 23, 22, 19 - AverageThreadTokens: 22.32 - PeakMemoryUsage: 744.39 MB (780546048) - PerHostPeakMemUsage: 750.66 MB (787128320) - PrepareTime: 204.122ms - RowsProduced: 18 - TotalCpuTime: 3m27s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 4.48ms - TotalStorageWaitTime: 1s241ms CodeGen:(Total: 468.382ms, non-child: 468.382ms, % non-child: 100.00%) - CodegenTime: 2.732ms - CompileTime: 41.537ms - LoadTime: 84.656us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 226.162ms - PrepareTime: 199.156ms DataStreamSender (dst_id=3):(Total: 35.247us, non-child: 35.247us, % non-child: 100.00%) - BytesSent: 222.00 B (222) - NetworkThroughput(*): 150.70 KB/sec - OverallThroughput: 6.01 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 39.981us - ThriftTransmitTime(*): 1.438ms - UncompressedRowBatchSize: 232.00 B (232) AGGREGATION_NODE (id=1):(Total: 9s168ms, non-child: 7s908ms, % non-child: 86.26%) ExecOption: Codegen Enabled - BuildTime: 7s487ms - GetNewBlockTime: 147.103us - GetResultsTime: 24.430us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 1.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.316us HDFS_SCAN_NODE (id=0):(Total: 1s259ms, non-child: 1s259ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB Hdfs Read Thread Concurrency Bucket: 0:66.67% 1:16.67% 2:16.67% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:64 ExecOption: Codegen enabled: 64 out of 64 BytesRead(500.0ms): 712.19 MB, 856.19 MB, 1.13 GB, 1.43 GB, 1.69 GB, 2.07 GB, 2.37 GB, 2.70 GB, 2.89 GB, 3.27 GB, 3.68 GB, 3.97 GB, 4.29 GB, 4.59 GB, 4.93 GB, 5.23 GB, 5.54 GB, 5.65 GB - AverageHdfsReadThreadConcurrency: 0.50 - AverageScannerThreadConcurrency: 22.50 - BytesRead: 5.66 GB (6079010449) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.66 GB (6079009425) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.66 GB (6079009425) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 738.06 MB (773910528) - PerReadThreadRawHdfsThroughput: 1.61 GB/sec - RemoteScanRanges: 2 - RowsRead: 260.29M (260289143) - RowsReturned: 260.29M (260289143) - RowsReturnedRate: 206.62 M/sec - ScanRangesComplete: 64 - ScannerThreadsInvoluntaryContextSwitches: 8.92K (8918) - ScannerThreadsTotalWallClockTime: 3m19s - DelimiterParseTime: 14s390ms - MaterializeTupleTime(*): 9s917ms - ScannerThreadsSysTime: 1s249ms - ScannerThreadsUserTime: 28s399ms - ScannerThreadsVoluntaryContextSwitches: 274.81K (274808) - TotalRawHdfsReadTime(*): 3s524ms - TotalReadThroughput: 643.26 MB/sec Instance ea4d4a38671669c3:4a4814fc2ba66ac0 (host=a1730.halxg.cloudera.com:22000):(Total: 9s089ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB MemoryUsage(500.0ms): 6.32 MB, 368.12 MB, 362.17 MB, 360.14 MB, 360.17 MB, 320.17 MB, 352.14 MB, 392.17 MB, 352.14 MB, 376.14 MB, 360.14 MB, 336.14 MB, 328.17 MB, 344.14 MB, 328.14 MB, 340.14 MB, 296.09 MB, 208.03 MB ThreadUsage(500.0ms): 1, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 14, 11 - AverageThreadTokens: 14.72 - PeakMemoryUsage: 488.17 MB (511881216) - PerHostPeakMemUsage: 494.46 MB (518475776) - PrepareTime: 200.323ms - RowsProduced: 18 - TotalCpuTime: 2m14s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 966.444us - TotalStorageWaitTime: 1s016ms CodeGen:(Total: 481.258ms, non-child: 481.258ms, % non-child: 100.00%) - CodegenTime: 2.725ms - CompileTime: 42.966ms - LoadTime: 87.236us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 241.437ms - PrepareTime: 195.302ms DataStreamSender (dst_id=3):(Total: 35.258us, non-child: 35.258us, % non-child: 100.00%) - BytesSent: 222.00 B (222) - NetworkThroughput(*): 253.58 KB/sec - OverallThroughput: 6.00 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 34.390us - ThriftTransmitTime(*): 854.940us - UncompressedRowBatchSize: 232.00 B (232) AGGREGATION_NODE (id=1):(Total: 9s088ms, non-child: 7s622ms, % non-child: 83.88%) ExecOption: Codegen Enabled - BuildTime: 7s178ms - GetNewBlockTime: 219.624us - GetResultsTime: 25.848us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 1.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.17us HDFS_SCAN_NODE (id=0):(Total: 1s465ms, non-child: 1s465ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB Hdfs Read Thread Concurrency Bucket: 0:76.47% 1:23.53% 2:0% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:59 ExecOption: Codegen enabled: 59 out of 59 BytesRead(500.0ms): 0, 574.79 MB, 904.31 MB, 1.21 GB, 1.54 GB, 1.81 GB, 2.18 GB, 2.53 GB, 2.78 GB, 3.09 GB, 3.37 GB, 3.68 GB, 3.98 GB, 4.33 GB, 4.63 GB, 4.98 GB, 5.24 GB, 5.49 GB - AverageHdfsReadThreadConcurrency: 0.24 - AverageScannerThreadConcurrency: 14.53 - BytesRead: 5.60 GB (6012072860) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.60 GB (6012070812) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.60 GB (6012070812) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 15 - PeakMemoryUsage: 481.84 MB (505245696) - PerReadThreadRawHdfsThroughput: 1.57 GB/sec - RemoteScanRanges: 4 - RowsRead: 257.43M (257426663) - RowsReturned: 257.43M (257426663) - RowsReturnedRate: 175.66 M/sec - ScanRangesComplete: 59 - ScannerThreadsInvoluntaryContextSwitches: 11.46K (11460) - ScannerThreadsTotalWallClockTime: 2m6s - DelimiterParseTime: 13s759ms - MaterializeTupleTime(*): 9s721ms - ScannerThreadsSysTime: 1s450ms - ScannerThreadsUserTime: 27s586ms - ScannerThreadsVoluntaryContextSwitches: 281.82K (281820) - TotalRawHdfsReadTime(*): 3s561ms - TotalReadThroughput: 624.62 MB/sec Instance ea4d4a38671669c3:4a4814fc2ba66abd (host=a1731.halxg.cloudera.com:22000):(Total: 8s507ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB MemoryUsage(500.0ms): 6.32 MB, 568.39 MB, 568.39 MB, 560.39 MB, 560.32 MB, 560.39 MB, 552.36 MB, 536.39 MB, 384.39 MB, 608.39 MB, 544.36 MB, 560.39 MB, 560.39 MB, 552.39 MB, 544.39 MB, 472.33 MB, 296.23 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 22, 19 - AverageThreadTokens: 22.24 - PeakMemoryUsage: 736.39 MB (772157440) - PerHostPeakMemUsage: 742.66 MB (778739712) - PrepareTime: 203.670ms - RowsProduced: 18 - TotalCpuTime: 3m11s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 3.599ms - TotalStorageWaitTime: 1s350ms CodeGen:(Total: 470.639ms, non-child: 470.639ms, % non-child: 100.00%) - CodegenTime: 2.658ms - CompileTime: 41.663ms - LoadTime: 57.590us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 228.550ms - PrepareTime: 199.71ms DataStreamSender (dst_id=3):(Total: 40.697us, non-child: 40.697us, % non-child: 100.00%) - BytesSent: 222.00 B (222) - NetworkThroughput(*): 153.10 KB/sec - OverallThroughput: 5.20 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 36.183us - ThriftTransmitTime(*): 1.416ms - UncompressedRowBatchSize: 232.00 B (232) AGGREGATION_NODE (id=1):(Total: 8s506ms, non-child: 7s171ms, % non-child: 84.30%) ExecOption: Codegen Enabled - BuildTime: 6s741ms - GetNewBlockTime: 110.70us - GetResultsTime: 20.740us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 1.956us HDFS_SCAN_NODE (id=0):(Total: 1s335ms, non-child: 1s335ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB Hdfs Read Thread Concurrency Bucket: 0:62.5% 1:31.25% 2:0% 3:6.25% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:53 ExecOption: Codegen enabled: 53 out of 53 BytesRead(500.0ms): 0, 731.13 MB, 1.07 GB, 1.41 GB, 1.73 GB, 2.07 GB, 2.39 GB, 2.75 GB, 2.95 GB, 3.48 GB, 3.77 GB, 4.13 GB, 4.48 GB, 4.79 GB, 5.14 GB, 5.45 GB, 5.64 GB - AverageHdfsReadThreadConcurrency: 0.50 - AverageScannerThreadConcurrency: 22.56 - BytesRead: 5.67 GB (6085373526) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.67 GB (6085367382) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.67 GB (6085367382) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 730.06 MB (765521920) - PerReadThreadRawHdfsThroughput: 1.54 GB/sec - RemoteScanRanges: 12 - RowsRead: 260.57M (260565882) - RowsReturned: 260.57M (260565882) - RowsReturnedRate: 195.14 M/sec - ScanRangesComplete: 53 - ScannerThreadsInvoluntaryContextSwitches: 9.33K (9328) - ScannerThreadsTotalWallClockTime: 3m4s - DelimiterParseTime: 14s767ms - MaterializeTupleTime(*): 10s253ms - ScannerThreadsSysTime: 1s322ms - ScannerThreadsUserTime: 28s904ms - ScannerThreadsVoluntaryContextSwitches: 280.83K (280828) - TotalRawHdfsReadTime(*): 3s677ms - TotalReadThroughput: 679.50 MB/sec Instance ea4d4a38671669c3:4a4814fc2ba66abe (host=a1733.halxg.cloudera.com:22000):(Total: 7s400ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB MemoryUsage(500.0ms): 0, 668.36 MB, 546.36 MB, 544.36 MB, 560.36 MB, 560.39 MB, 548.36 MB, 560.51 MB, 544.39 MB, 336.39 MB, 432.25 MB, 392.22 MB, 368.17 MB, 360.17 MB, 320.10 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 19, 18, 16, 16, 14 - AverageThreadTokens: 20.00 - PeakMemoryUsage: 672.39 MB (705048576) - PerHostPeakMemUsage: 678.66 MB (711630848) - PrepareTime: 200.326ms - RowsProduced: 18 - TotalCpuTime: 2m30s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 4.329ms - TotalStorageWaitTime: 1s721ms CodeGen:(Total: 465.767ms, non-child: 465.767ms, % non-child: 100.00%) - CodegenTime: 2.685ms - CompileTime: 41.204ms - LoadTime: 72.809us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 227.708ms - PrepareTime: 195.449ms DataStreamSender (dst_id=3):(Total: 41.23us, non-child: 41.23us, % non-child: 100.00%) - BytesSent: 222.00 B (222) - NetworkThroughput(*): 119.99 KB/sec - OverallThroughput: 5.16 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 36.927us - ThriftTransmitTime(*): 1.806ms - UncompressedRowBatchSize: 232.00 B (232) AGGREGATION_NODE (id=1):(Total: 7s399ms, non-child: 6s248ms, % non-child: 84.44%) ExecOption: Codegen Enabled - BuildTime: 5s852ms - GetNewBlockTime: 133.363us - GetResultsTime: 25.183us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.39us HDFS_SCAN_NODE (id=0):(Total: 1s151ms, non-child: 1s151ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB Hdfs Read Thread Concurrency Bucket: 0:64.29% 1:35.71% 2:0% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:94 ExecOption: Codegen enabled: 94 out of 94 BytesRead(500.0ms): 670.81 MB, 907.59 MB, 1.23 GB, 1.57 GB, 1.90 GB, 2.22 GB, 2.55 GB, 2.90 GB, 3.05 GB, 3.49 GB, 3.80 GB, 4.12 GB, 4.44 GB, 4.75 GB - AverageHdfsReadThreadConcurrency: 0.36 - AverageScannerThreadConcurrency: 20.36 - BytesRead: 4.82 GB (5178211951) - BytesReadDataNodeCache: 0 - BytesReadLocal: 4.82 GB (5178198639) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 4.82 GB (5178198639) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 666.06 MB (698413056) - PerReadThreadRawHdfsThroughput: 1.55 GB/sec - RemoteScanRanges: 26 - RowsRead: 221.72M (221718312) - RowsReturned: 221.72M (221718312) - RowsReturnedRate: 192.60 M/sec - ScanRangesComplete: 94 - ScannerThreadsInvoluntaryContextSwitches: 7.65K (7650) - ScannerThreadsTotalWallClockTime: 2m24s - DelimiterParseTime: 11s921ms - MaterializeTupleTime(*): 8s431ms - ScannerThreadsSysTime: 1s113ms - ScannerThreadsUserTime: 23s739ms - ScannerThreadsVoluntaryContextSwitches: 240.80K (240802) - TotalRawHdfsReadTime(*): 3s113ms - TotalReadThroughput: 695.19 MB/sec [localhost:21000] > select count(distinct y) from billion_rows_text; +-------------------+ | count(distinct y) | +-------------------+ | 19 | +-------------------+ [localhost:21000] > profile; Query Runtime Profile: Query (id=e0420f92157bd791:70b4040e80dba1b4): Summary: Session ID: 8f4cad0b2b385c70:b41743dae24d22ad Session Type: BEESWAX Start Time: 2015-01-12 15:14:19.512054000 End Time: 2015-01-12 15:14:29.195633000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) User: jrussell Connected User: jrussell Delegated User: Network Address: 127.0.0.1:38627 Default Db: oreilly Sql Statement: select count(distinct y) from billion_rows_text Coordinator: a1730.halxg.cloudera.com:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=3.88GB VCores=2 WARNING: The following tables are missing relevant table and/or column statistics. oreilly.billion_rows_text F02:PLAN FRAGMENT [UNPARTITIONED] 06:AGGREGATE [FINALIZE] | output: count:merge(y) | hosts=4 per-host-mem=unavailable | tuple-ids=2 row-size=8B cardinality=1 | 05:EXCHANGE [UNPARTITIONED] hosts=4 per-host-mem=unavailable tuple-ids=2 row-size=8B cardinality=1 F01:PLAN FRAGMENT [HASH(y)] DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=05, UNPARTITIONED] 02:AGGREGATE | output: count(y) | hosts=4 per-host-mem=10.00MB | tuple-ids=2 row-size=8B cardinality=1 | 04:AGGREGATE | group by: y | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=4B cardinality=unavailable | 03:EXCHANGE [HASH(y)] hosts=4 per-host-mem=0B tuple-ids=1 row-size=4B cardinality=unavailable F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=03, HASH(y)] 01:AGGREGATE | group by: y | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=4B cardinality=unavailable | 00:SCAN HDFS [oreilly.billion_rows_text, RANDOM] partitions=1/1 files=113 size=21.75GB table stats: unavailable column stats: unavailable hosts=4 per-host-mem=3.75GB tuple-ids=0 row-size=4B cardinality=unavailable ---------------- Estimated Per-Host Mem: 4160749568 Estimated Per-Host VCores: 2 Tables Missing Stats: oreilly.billion_rows_text Admission result: Admitted immediately Request Pool: root.jrussell ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail -------------------------------------------------------------------------------------------------------------------- 06:AGGREGATE 1 196.521ms 196.521ms 1 1 28.00 KB -1.00 B FINALIZE 05:EXCHANGE 1 42.173us 42.173us 4 1 0 -1.00 B UNPARTITIONED 02:AGGREGATE 4 193.682ms 195.369ms 4 1 12.00 KB 10.00 MB 04:AGGREGATE 4 2.715ms 2.801ms 19 -1 6.27 MB 128.00 MB 03:EXCHANGE 4 37.794us 52.45us 76 -1 0 0 HASH(y) 01:AGGREGATE 4 7s071ms 7s570ms 76 -1 6.29 MB 128.00 MB 00:SCAN HDFS 4 1s327ms 1s444ms 1.00B -1 730.06 MB 3.75 GB oreilly.billion_rows_text Query Timeline: 9s687ms - Start execution: 62.634us (62.634us) - Planning finished: 25.945ms (25.882ms) - Submit for admission: 26.482ms (536.890us) - Completed admission: 26.549ms (67.415us) - Ready to start remote fragments: 223.785ms (197.235ms) - Remote fragments started: 630.385ms (406.600ms) - Rows available: 9s678ms (9s048ms) - First row fetched: 9s681ms (2.100ms) - Unregister query: 9s683ms (2.483ms) ImpalaServer: - ClientFetchWaitTimer: 3.445ms - RowMaterializationTimer: 5.561us Execution Profile e0420f92157bd791:70b4040e80dba1b4:(Total: 9s652ms, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 8, last: 0.205652, min: 0.195806, max: 0.205652, mean: 0.200268, stddev: 0.00334825 Per Node Peak Memory Usage: a1733.halxg.cloudera.com:22000(690.67 MB) a1731.halxg.cloudera.com:22000(742.66 MB) a1732.halxg.cloudera.com:22000(722.66 MB) a1730.halxg.cloudera.com:22000(478.46 MB) - FinalizationTimer: 0ns Coordinator Fragment F02:(Total: 9s048ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB ThreadUsage(500.0ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 40.00 KB (40960) - PerHostPeakMemUsage: 0 - PrepareTime: 196.451ms - RowsProduced: 1 - TotalCpuTime: 802.241ms - TotalNetworkReceiveTime: 8s851ms - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 392.272ms, non-child: 392.272ms, % non-child: 100.00%) - CodegenTime: 530.885us - CompileTime: 6.679ms - LoadTime: 35.34us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 189.647ms - PrepareTime: 194.391ms AGGREGATION_NODE (id=6):(Total: 9s048ms, non-child: 196.521ms, % non-child: 2.17%) ExecOption: Codegen Enabled - BuildTime: 3.390us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 28.00 KB (28672) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 EXCHANGE_NODE (id=5):(Total: 8s851ms, non-child: 42.173us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 64.00 B (64) - ConvertRowBatchTime: 6.536us - DeserializeRowBatchTimer: 19.972us - FirstBatchArrivalWaitTime: 8s848ms - PeakMemoryUsage: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F01:(Total: 9s207ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:9s254ms max:9s257ms mean: 9s256ms stddev:1.85ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 4 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6604800) - PerHostPeakMemUsage: 658.61 MB (690605312) - PrepareTime: 195.951ms - RowsProduced: 1 - TotalCpuTime: 442.13ms - TotalNetworkReceiveTime: 9s010ms - TotalNetworkSendTime: 478.686us - TotalStorageWaitTime: 0ns CodeGen:(Total: 437.415ms, non-child: 437.415ms, % non-child: 100.00%) - CodegenTime: 2.264ms - CompileTime: 30.868ms - LoadTime: 79.436us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 213.222ms - PrepareTime: 191.889ms DataStreamSender (dst_id=5):(Total: 32.841us, non-child: 32.841us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 70.12 KB/sec - OverallThroughput: 478.03 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 4.729us - ThriftTransmitTime(*): 225.970us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s206ms, non-child: 193.682ms, % non-child: 2.10%) - BuildTime: 6.342us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s012ms, non-child: 2.715ms, % non-child: 0.03%) - BuildTime: 9.833us - GetNewBlockTime: 85.690us - GetResultsTime: 8.217us - HashBuckets: 4.61K (4608) - LargestPartitionPercent: 4 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6572032) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.109us EXCHANGE_NODE (id=3):(Total: 9s010ms, non-child: 9s010ms, % non-child: 100.00%) - BytesReceived: 219.00 B (219) - ConvertRowBatchTime: 6.562us - DeserializeRowBatchTimer: 24.485us - FirstBatchArrivalWaitTime: 7s691ms - PeakMemoryUsage: 0 - RowsReturned: 19 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns Averaged Fragment F00:(Total: 8s400ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 4.82 GB, max: 5.67 GB, avg: 5.44 GB, stddev: 364.68 MB completion times: min:7s293ms max:9s048ms mean: 8s474ms stddev:702.761ms execution rates: min:633.67 MB/sec max:677.12 MB/sec mean:658.24 MB/sec stddev:18.28 MB/sec num instances: 4 - AverageThreadTokens: 19.41 - PeakMemoryUsage: 652.33 MB (684019968) - PerHostPeakMemUsage: 658.61 MB (690605312) - PrepareTime: 201.251ms - RowsProduced: 19 - TotalCpuTime: 2m47s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.20ms - TotalStorageWaitTime: 1s555ms CodeGen:(Total: 470.167ms, non-child: 470.167ms, % non-child: 100.00%) - CodegenTime: 2.656ms - CompileTime: 41.690ms - LoadTime: 73.336us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 230.519ms - PrepareTime: 196.497ms DataStreamSender (dst_id=3):(Total: 38.431us, non-child: 38.431us, % non-child: 100.00%) - BytesSent: 219.00 B (219) - NetworkThroughput(*): 275.01 KB/sec - OverallThroughput: 5.46 MB/sec - PeakMemoryUsage: 160.00 KB (163840) - SerializeBatchTime: 31.75us - ThriftTransmitTime(*): 781.891us - UncompressedRowBatchSize: 240.00 B (240) AGGREGATION_NODE (id=1):(Total: 8s399ms, non-child: 7s071ms, % non-child: 84.19%) - BuildTime: 6s647ms - GetNewBlockTime: 111.33us - GetResultsTime: 25.95us - HashBuckets: 16.38K (16384) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.27us HDFS_SCAN_NODE (id=0):(Total: 1s327ms, non-child: 1s327ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.46 - AverageScannerThreadConcurrency: 19.55 - BytesRead: 5.44 GB (5838667196) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.44 GB (5838661564) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.44 GB (5838661564) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 21 - PeakMemoryUsage: 646.00 MB (677378304) - PerReadThreadRawHdfsThroughput: 1.56 GB/sec - RemoteScanRanges: 16 - RowsRead: 250.00M (250000000) - RowsReturned: 250.00M (250000000) - RowsReturnedRate: 188.82 M/sec - ScanRangesComplete: 67 - ScannerThreadsInvoluntaryContextSwitches: 8.94K (8944) - ScannerThreadsTotalWallClockTime: 2m39s - DelimiterParseTime: 13s770ms - MaterializeTupleTime(*): 8s458ms - ScannerThreadsSysTime: 1s368ms - ScannerThreadsUserTime: 26s145ms - ScannerThreadsVoluntaryContextSwitches: 269.13K (269135) - TotalRawHdfsReadTime(*): 3s478ms - TotalReadThroughput: 663.01 MB/sec Fragment F01: Instance e0420f92157bd791:70b4040e80dba1b8 (host=a1732.halxg.cloudera.com:22000):(Total: 9s210ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.29 MB (6590464) - PerHostPeakMemUsage: 722.66 MB (757768192) - PrepareTime: 197.684ms - RowsProduced: 1 - TotalCpuTime: 440.809ms - TotalNetworkReceiveTime: 9s011ms - TotalNetworkSendTime: 291.360us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 436.8ms, non-child: 436.8ms, % non-child: 100.00%) - CodegenTime: 2.289ms - CompileTime: 30.428ms - LoadTime: 82.479us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 210.569ms - PrepareTime: 193.442ms DataStreamSender (dst_id=5):(Total: 32.401us, non-child: 32.401us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 77.35 KB/sec - OverallThroughput: 482.24 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.744us - ThriftTransmitTime(*): 202.9us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s209ms, non-child: 195.369ms, % non-child: 2.12%) ExecOption: Codegen Enabled - BuildTime: 1.70us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s014ms, non-child: 2.724ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 2.388us - GetNewBlockTime: 87.703us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.26 MB (6561792) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.214us EXCHANGE_NODE (id=3):(Total: 9s011ms, non-child: 6.8us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 0 - ConvertRowBatchTime: 695ns - DeserializeRowBatchTimer: 0ns - FirstBatchArrivalWaitTime: 9s011ms - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance e0420f92157bd791:70b4040e80dba1b7 (host=a1733.halxg.cloudera.com:22000):(Total: 9s206ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.29 MB, 6.29 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6606848) - PerHostPeakMemUsage: 690.67 MB (724214784) - PrepareTime: 195.865ms - RowsProduced: 1 - TotalCpuTime: 442.724ms - TotalNetworkReceiveTime: 9s009ms - TotalNetworkSendTime: 322.549us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 438.10ms, non-child: 438.10ms, % non-child: 100.00%) - CodegenTime: 2.239ms - CompileTime: 30.899ms - LoadTime: 69.76us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 213.847ms - PrepareTime: 191.806ms DataStreamSender (dst_id=5):(Total: 34.705us, non-child: 34.705us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 75.18 KB/sec - OverallThroughput: 450.22 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 6.955us - ThriftTransmitTime(*): 207.843us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s205ms, non-child: 193.626ms, % non-child: 2.10%) ExecOption: Codegen Enabled - BuildTime: 8.955us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s012ms, non-child: 2.722ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 17.48us - GetNewBlockTime: 84.635us - GetResultsTime: 7.934us - HashBuckets: 5.12K (5120) - LargestPartitionPercent: 5 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6569984) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.28us EXCHANGE_NODE (id=3):(Total: 9s009ms, non-child: 42.938us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60.00 B, 60.00 B, 60.00 B, 180.00 B - BytesReceived: 240.00 B (240) - ConvertRowBatchTime: 7.511us - DeserializeRowBatchTimer: 32.131us - FirstBatchArrivalWaitTime: 7s251ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance e0420f92157bd791:70b4040e80dba1b6 (host=a1731.halxg.cloudera.com:22000):(Total: 9s206ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 742.66 MB (778739712) - PrepareTime: 194.472ms - RowsProduced: 1 - TotalCpuTime: 440.536ms - TotalNetworkReceiveTime: 9s010ms - TotalNetworkSendTime: 328.292us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 436.9ms, non-child: 436.9ms, % non-child: 100.00%) - CodegenTime: 2.199ms - CompileTime: 30.893ms - LoadTime: 70.789us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 213.174ms - PrepareTime: 190.522ms DataStreamSender (dst_id=5):(Total: 29.411us, non-child: 29.411us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 57.21 KB/sec - OverallThroughput: 531.26 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.870us - ThriftTransmitTime(*): 273.109us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s205ms, non-child: 192.279ms, % non-child: 2.09%) ExecOption: Codegen Enabled - BuildTime: 6.594us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s013ms, non-child: 2.613ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 7.327us - GetNewBlockTime: 73.544us - GetResultsTime: 5.20us - HashBuckets: 3.07K (3072) - LargestPartitionPercent: 8 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 3 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 1.946us EXCHANGE_NODE (id=3):(Total: 9s010ms, non-child: 50.188us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 40.00 B, 40.00 B, 80.00 B - BytesReceived: 160.00 B (160) - ConvertRowBatchTime: 7.503us - DeserializeRowBatchTimer: 26.230us - FirstBatchArrivalWaitTime: 7s253ms - PeakMemoryUsage: 0 - RowsReturned: 12 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance e0420f92157bd791:70b4040e80dba1b9 (host=a1730.halxg.cloudera.com:22000):(Total: 9s205ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 478.46 MB (501698560) - PrepareTime: 195.783ms - RowsProduced: 1 - TotalCpuTime: 443.982ms - TotalNetworkReceiveTime: 9s008ms - TotalNetworkSendTime: 972.546us - TotalStorageWaitTime: 0ns CodeGen:(Total: 439.630ms, non-child: 439.630ms, % non-child: 100.00%) - CodegenTime: 2.327ms - CompileTime: 31.254ms - LoadTime: 95.400us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 215.298ms - PrepareTime: 191.784ms DataStreamSender (dst_id=5):(Total: 34.847us, non-child: 34.847us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 70.73 KB/sec - OverallThroughput: 448.39 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 4.350us - ThriftTransmitTime(*): 220.921us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s205ms, non-child: 193.455ms, % non-child: 2.10%) ExecOption: Codegen Enabled - BuildTime: 8.750us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s011ms, non-child: 2.801ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 12.570us - GetNewBlockTime: 96.880us - GetResultsTime: 19.916us - HashBuckets: 10.24K (10240) - LargestPartitionPercent: 4 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 11 - RowsReturnedRate: 1.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.251us EXCHANGE_NODE (id=3):(Total: 9s008ms, non-child: 52.45us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 119.00 B, 119.00 B, 238.00 B - BytesReceived: 476.00 B (476) - ConvertRowBatchTime: 10.542us - DeserializeRowBatchTimer: 39.582us - FirstBatchArrivalWaitTime: 7s250ms - PeakMemoryUsage: 0 - RowsReturned: 44 - RowsReturnedRate: 4.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Fragment F00: Instance e0420f92157bd791:70b4040e80dba1bd (host=a1730.halxg.cloudera.com:22000):(Total: 8s966ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB MemoryUsage(500.0ms): 6.32 MB, 376.14 MB, 368.17 MB, 368.17 MB, 360.14 MB, 376.17 MB, 288.17 MB, 376.14 MB, 370.14 MB, 376.14 MB, 352.17 MB, 328.14 MB, 304.17 MB, 368.17 MB, 368.17 MB, 356.17 MB, 296.06 MB, 151.96 MB ThreadUsage(500.0ms): 1, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 13, 9 - AverageThreadTokens: 14.61 - PeakMemoryUsage: 472.17 MB (495104000) - PerHostPeakMemUsage: 478.46 MB (501698560) - PrepareTime: 204.182ms - RowsProduced: 19 - TotalCpuTime: 2m13s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.120ms - TotalStorageWaitTime: 1s013ms CodeGen:(Total: 482.656ms, non-child: 482.656ms, % non-child: 100.00%) - CodegenTime: 2.823ms - CompileTime: 42.595ms - LoadTime: 73.739us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 239.392ms - PrepareTime: 199.153ms DataStreamSender (dst_id=3):(Total: 41.750us, non-child: 41.750us, % non-child: 100.00%) - BytesSent: 219.00 B (219) - NetworkThroughput(*): 250.89 KB/sec - OverallThroughput: 5.00 MB/sec - PeakMemoryUsage: 160.00 KB (163840) - SerializeBatchTime: 34.468us - ThriftTransmitTime(*): 852.431us - UncompressedRowBatchSize: 240.00 B (240) AGGREGATION_NODE (id=1):(Total: 8s965ms, non-child: 7s521ms, % non-child: 83.89%) ExecOption: Codegen Enabled - BuildTime: 7s077ms - GetNewBlockTime: 122.757us - GetResultsTime: 28.330us - HashBuckets: 16.38K (16384) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.179us HDFS_SCAN_NODE (id=0):(Total: 1s444ms, non-child: 1s444ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB Hdfs Read Thread Concurrency Bucket: 0:52.94% 1:35.29% 2:11.76% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:59 ExecOption: Codegen enabled: 59 out of 59 BytesRead(500.0ms): 0, 573.35 MB, 869.35 MB, 1.18 GB, 1.49 GB, 1.82 GB, 2.08 GB, 2.49 GB, 2.80 GB, 3.11 GB, 3.43 GB, 3.74 GB, 4.03 GB, 4.42 GB, 4.74 GB, 5.04 GB, 5.34 GB, 5.54 GB - AverageHdfsReadThreadConcurrency: 0.59 - AverageScannerThreadConcurrency: 14.41 - BytesRead: 5.60 GB (6012072860) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.60 GB (6012070812) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.60 GB (6012070812) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 15 - PeakMemoryUsage: 465.84 MB (488468480) - PerReadThreadRawHdfsThroughput: 1.60 GB/sec - RemoteScanRanges: 6 - RowsRead: 257.43M (257426663) - RowsReturned: 257.43M (257426663) - RowsReturnedRate: 178.23 M/sec - ScanRangesComplete: 59 - ScannerThreadsInvoluntaryContextSwitches: 10.58K (10580) - ScannerThreadsTotalWallClockTime: 2m5s - DelimiterParseTime: 14s452ms - MaterializeTupleTime(*): 8s730ms - ScannerThreadsSysTime: 1s354ms - ScannerThreadsUserTime: 27s594ms - ScannerThreadsVoluntaryContextSwitches: 281.85K (281851) - TotalRawHdfsReadTime(*): 3s502ms - TotalReadThroughput: 629.95 MB/sec Instance e0420f92157bd791:70b4040e80dba1ba (host=a1731.halxg.cloudera.com:22000):(Total: 8s888ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB MemoryUsage(500.0ms): 6.32 MB, 552.39 MB, 568.39 MB, 560.36 MB, 560.36 MB, 560.39 MB, 544.39 MB, 544.39 MB, 408.36 MB, 560.36 MB, 544.39 MB, 560.39 MB, 556.39 MB, 568.37 MB, 544.39 MB, 508.36 MB, 424.23 MB, 176.11 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 19, 14 - AverageThreadTokens: 21.89 - PeakMemoryUsage: 736.39 MB (772157440) - PerHostPeakMemUsage: 742.66 MB (778739712) - PrepareTime: 198.863ms - RowsProduced: 19 - TotalCpuTime: 3m18s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.8ms - TotalStorageWaitTime: 1s284ms CodeGen:(Total: 463.848ms, non-child: 463.848ms, % non-child: 100.00%) - CodegenTime: 2.558ms - CompileTime: 41.344ms - LoadTime: 60.931us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 226.731ms - PrepareTime: 194.356ms DataStreamSender (dst_id=3):(Total: 34.836us, non-child: 34.836us, % non-child: 100.00%) - BytesSent: 219.00 B (219) - NetworkThroughput(*): 278.20 KB/sec - OverallThroughput: 6.00 MB/sec - PeakMemoryUsage: 160.00 KB (163840) - SerializeBatchTime: 29.866us - ThriftTransmitTime(*): 768.743us - UncompressedRowBatchSize: 240.00 B (240) AGGREGATION_NODE (id=1):(Total: 8s887ms, non-child: 7s570ms, % non-child: 85.19%) ExecOption: Codegen Enabled - BuildTime: 7s143ms - GetNewBlockTime: 105.707us - GetResultsTime: 26.628us - HashBuckets: 16.38K (16384) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.6us HDFS_SCAN_NODE (id=0):(Total: 1s316ms, non-child: 1s316ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB Hdfs Read Thread Concurrency Bucket: 0:64.71% 1:29.41% 2:5.882% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:53 ExecOption: Codegen enabled: 53 out of 53 BytesRead(500.0ms): 0, 728.22 MB, 1.07 GB, 1.39 GB, 1.73 GB, 2.05 GB, 2.34 GB, 2.68 GB, 2.89 GB, 3.37 GB, 3.68 GB, 4.00 GB, 4.34 GB, 4.68 GB, 4.99 GB, 5.26 GB, 5.53 GB, 5.62 GB - AverageHdfsReadThreadConcurrency: 0.41 - AverageScannerThreadConcurrency: 22.12 - BytesRead: 5.67 GB (6085373526) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.67 GB (6085367382) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.67 GB (6085367382) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 730.06 MB (765521920) - PerReadThreadRawHdfsThroughput: 1.56 GB/sec - RemoteScanRanges: 18 - RowsRead: 260.57M (260565882) - RowsReturned: 260.57M (260565882) - RowsReturnedRate: 197.93 M/sec - ScanRangesComplete: 53 - ScannerThreadsInvoluntaryContextSwitches: 8.99K (8994) - ScannerThreadsTotalWallClockTime: 3m10s - DelimiterParseTime: 14s643ms - MaterializeTupleTime(*): 8s842ms - ScannerThreadsSysTime: 1s251ms - ScannerThreadsUserTime: 27s522ms - ScannerThreadsVoluntaryContextSwitches: 276.71K (276710) - TotalRawHdfsReadTime(*): 3s634ms - TotalReadThroughput: 639.49 MB/sec Instance e0420f92157bd791:70b4040e80dba1bc (host=a1732.halxg.cloudera.com:22000):(Total: 8s524ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB MemoryUsage(500.0ms): 0, 652.39 MB, 552.36 MB, 560.36 MB, 546.39 MB, 552.39 MB, 560.37 MB, 560.39 MB, 496.36 MB, 338.39 MB, 576.36 MB, 568.39 MB, 552.36 MB, 552.39 MB, 528.36 MB, 500.33 MB, 424.28 MB, 55.87 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 23, 22, 20, 5 - AverageThreadTokens: 21.28 - PeakMemoryUsage: 716.39 MB (751185920) - PerHostPeakMemUsage: 722.66 MB (757768192) - PrepareTime: 201.359ms - RowsProduced: 19 - TotalCpuTime: 3m9s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.103ms - TotalStorageWaitTime: 1s442ms CodeGen:(Total: 467.587ms, non-child: 467.587ms, % non-child: 100.00%) - CodegenTime: 2.637ms - CompileTime: 41.457ms - LoadTime: 83.52us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 228.29ms - PrepareTime: 196.684ms DataStreamSender (dst_id=3):(Total: 36.741us, non-child: 36.741us, % non-child: 100.00%) - BytesSent: 219.00 B (219) - NetworkThroughput(*): 264.58 KB/sec - OverallThroughput: 5.68 MB/sec - PeakMemoryUsage: 160.00 KB (163840) - SerializeBatchTime: 30.867us - ThriftTransmitTime(*): 808.311us - UncompressedRowBatchSize: 240.00 B (240) AGGREGATION_NODE (id=1):(Total: 8s523ms, non-child: 7s112ms, % non-child: 83.44%) ExecOption: Codegen Enabled - BuildTime: 6s681ms - GetNewBlockTime: 112.917us - GetResultsTime: 20.331us - HashBuckets: 16.38K (16384) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 1.933us HDFS_SCAN_NODE (id=0):(Total: 1s411ms, non-child: 1s411ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB Hdfs Read Thread Concurrency Bucket: 0:70.59% 1:17.65% 2:11.76% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:64 ExecOption: Codegen enabled: 64 out of 64 BytesRead(500.0ms): 629.58 MB, 893.58 MB, 1.24 GB, 1.56 GB, 1.89 GB, 2.25 GB, 2.61 GB, 2.89 GB, 3.07 GB, 3.63 GB, 3.96 GB, 4.29 GB, 4.65 GB, 4.99 GB, 5.31 GB, 5.59 GB, 5.63 GB - AverageHdfsReadThreadConcurrency: 0.41 - AverageScannerThreadConcurrency: 21.47 - BytesRead: 5.66 GB (6079010449) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.66 GB (6079009425) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.66 GB (6079009425) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 710.06 MB (744550400) - PerReadThreadRawHdfsThroughput: 1.52 GB/sec - RemoteScanRanges: 3 - RowsRead: 260.29M (260289143) - RowsReturned: 260.29M (260289143) - RowsReturnedRate: 184.45 M/sec - ScanRangesComplete: 64 - ScannerThreadsInvoluntaryContextSwitches: 9.03K (9027) - ScannerThreadsTotalWallClockTime: 3m2s - DelimiterParseTime: 13s984ms - MaterializeTupleTime(*): 8s692ms - ScannerThreadsSysTime: 1s819ms - ScannerThreadsUserTime: 26s556ms - ScannerThreadsVoluntaryContextSwitches: 279.21K (279209) - TotalRawHdfsReadTime(*): 3s727ms - TotalReadThroughput: 678.28 MB/sec Instance e0420f92157bd791:70b4040e80dba1bb (host=a1733.halxg.cloudera.com:22000):(Total: 7s222ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB MemoryUsage(500.0ms): 0, 572.39 MB, 568.39 MB, 560.36 MB, 568.36 MB, 560.39 MB, 548.39 MB, 532.49 MB, 504.36 MB, 440.39 MB, 416.23 MB, 384.20 MB, 368.17 MB, 344.17 MB, 256.09 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 19, 17, 16, 16, 13 - AverageThreadTokens: 19.87 - PeakMemoryUsage: 684.39 MB (717632512) - PerHostPeakMemUsage: 690.67 MB (724214784) - PrepareTime: 200.601ms - RowsProduced: 19 - TotalCpuTime: 2m26s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 849.631us - TotalStorageWaitTime: 2s480ms CodeGen:(Total: 466.578ms, non-child: 466.578ms, % non-child: 100.00%) - CodegenTime: 2.606ms - CompileTime: 41.364ms - LoadTime: 75.624us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 227.925ms - PrepareTime: 195.794ms DataStreamSender (dst_id=3):(Total: 40.400us, non-child: 40.400us, % non-child: 100.00%) - BytesSent: 219.00 B (219) - NetworkThroughput(*): 306.36 KB/sec - OverallThroughput: 5.17 MB/sec - PeakMemoryUsage: 160.00 KB (163840) - SerializeBatchTime: 29.100us - ThriftTransmitTime(*): 698.81us - UncompressedRowBatchSize: 240.00 B (240) AGGREGATION_NODE (id=1):(Total: 7s221ms, non-child: 6s082ms, % non-child: 84.23%) ExecOption: Codegen Enabled - BuildTime: 5s688ms - GetNewBlockTime: 102.751us - GetResultsTime: 25.94us - HashBuckets: 16.38K (16384) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 1.990us HDFS_SCAN_NODE (id=0):(Total: 1s139ms, non-child: 1s139ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB Hdfs Read Thread Concurrency Bucket: 0:64.29% 1:28.57% 2:7.143% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:94 ExecOption: Codegen enabled: 94 out of 94 BytesRead(500.0ms): 578.08 MB, 936.66 MB, 1.26 GB, 1.61 GB, 1.94 GB, 2.29 GB, 2.60 GB, 2.94 GB, 3.23 GB, 3.57 GB, 3.89 GB, 4.20 GB, 4.53 GB, 4.81 GB - AverageHdfsReadThreadConcurrency: 0.43 - AverageScannerThreadConcurrency: 20.21 - BytesRead: 4.82 GB (5178211951) - BytesReadDataNodeCache: 0 - BytesReadLocal: 4.82 GB (5178198639) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 4.82 GB (5178198639) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 678.04 MB (710972416) - PerReadThreadRawHdfsThroughput: 1.58 GB/sec - RemoteScanRanges: 39 - RowsRead: 221.72M (221718312) - RowsReturned: 221.72M (221718312) - RowsReturnedRate: 194.66 M/sec - ScanRangesComplete: 94 - ScannerThreadsInvoluntaryContextSwitches: 7.18K (7178) - ScannerThreadsTotalWallClockTime: 2m21s - DelimiterParseTime: 12s001ms - MaterializeTupleTime(*): 7s567ms - ScannerThreadsSysTime: 1s045ms - ScannerThreadsUserTime: 22s908ms - ScannerThreadsVoluntaryContextSwitches: 238.77K (238771) - TotalRawHdfsReadTime(*): 3s048ms - TotalReadThroughput: 704.33 MB/sec [localhost:21000] > select count(distinct z) from billion_rows_text; +-------------------+ | count(distinct z) | +-------------------+ | 20 | +-------------------+ [localhost:21000] > profile; Query Runtime Profile: Query (id=8d42458318c7ceed:a414fa6edfa8a9a): Summary: Session ID: 8f4cad0b2b385c70:b41743dae24d22ad Session Type: BEESWAX Start Time: 2015-01-12 15:14:38.471234000 End Time: 2015-01-12 15:14:48.161291000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) User: jrussell Connected User: jrussell Delegated User: Network Address: 127.0.0.1:38627 Default Db: oreilly Sql Statement: select count(distinct z) from billion_rows_text Coordinator: a1730.halxg.cloudera.com:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=3.88GB VCores=2 WARNING: The following tables are missing relevant table and/or column statistics. oreilly.billion_rows_text F02:PLAN FRAGMENT [UNPARTITIONED] 06:AGGREGATE [FINALIZE] | output: count:merge(z) | hosts=4 per-host-mem=unavailable | tuple-ids=2 row-size=8B cardinality=1 | 05:EXCHANGE [UNPARTITIONED] hosts=4 per-host-mem=unavailable tuple-ids=2 row-size=8B cardinality=1 F01:PLAN FRAGMENT [HASH(z)] DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=05, UNPARTITIONED] 02:AGGREGATE | output: count(z) | hosts=4 per-host-mem=10.00MB | tuple-ids=2 row-size=8B cardinality=1 | 04:AGGREGATE | group by: z | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=4B cardinality=unavailable | 03:EXCHANGE [HASH(z)] hosts=4 per-host-mem=0B tuple-ids=1 row-size=4B cardinality=unavailable F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=03, HASH(z)] 01:AGGREGATE | group by: z | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=4B cardinality=unavailable | 00:SCAN HDFS [oreilly.billion_rows_text, RANDOM] partitions=1/1 files=113 size=21.75GB table stats: unavailable column stats: unavailable hosts=4 per-host-mem=3.75GB tuple-ids=0 row-size=4B cardinality=unavailable ---------------- Estimated Per-Host Mem: 4160749568 Estimated Per-Host VCores: 2 Tables Missing Stats: oreilly.billion_rows_text Admission result: Admitted immediately Request Pool: root.jrussell ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail -------------------------------------------------------------------------------------------------------------------- 06:AGGREGATE 1 200.279ms 200.279ms 1 1 24.00 KB -1.00 B FINALIZE 05:EXCHANGE 1 29.792us 29.792us 4 1 0 -1.00 B UNPARTITIONED 02:AGGREGATE 4 193.310ms 195.970ms 4 1 12.00 KB 10.00 MB 04:AGGREGATE 4 2.759ms 2.801ms 20 -1 6.27 MB 128.00 MB 03:EXCHANGE 4 80.432us 161.963us 80 -1 0 0 HASH(z) 01:AGGREGATE 4 6s918ms 7s562ms 80 -1 6.29 MB 128.00 MB 00:SCAN HDFS 4 1s319ms 1s536ms 1.00B -1 730.06 MB 3.75 GB oreilly.billion_rows_text Query Timeline: 9s694ms - Start execution: 57.892us (57.892us) - Planning finished: 26.663ms (26.606ms) - Submit for admission: 27.204ms (540.422us) - Completed admission: 27.264ms (60.332us) - Ready to start remote fragments: 228.119ms (200.854ms) - Remote fragments started: 636.838ms (408.719ms) - Rows available: 9s605ms (8s968ms) - First row fetched: 9s687ms (82.346ms) - Unregister query: 9s690ms (2.581ms) ImpalaServer: - ClientFetchWaitTimer: 83.767ms - RowMaterializationTimer: 5.982us Execution Profile 8d42458318c7ceed:a414fa6edfa8a9a:(Total: 9s578ms, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 8, last: 0.207781, min: 0.195588, max: 0.207781, mean: 0.200961, stddev: 0.00446657 Per Node Peak Memory Usage: a1733.halxg.cloudera.com:22000(664.66 MB) a1731.halxg.cloudera.com:22000(742.66 MB) a1732.halxg.cloudera.com:22000(710.66 MB) a1730.halxg.cloudera.com:22000(494.46 MB) - FinalizationTimer: 0ns Coordinator Fragment F02:(Total: 8s967ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB ThreadUsage(500.0ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 40.00 KB (40960) - PerHostPeakMemUsage: 0 - PrepareTime: 200.90ms - RowsProduced: 1 - TotalCpuTime: 892.389ms - TotalNetworkReceiveTime: 8s767ms - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 399.981ms, non-child: 399.981ms, % non-child: 100.00%) - CodegenTime: 513.213us - CompileTime: 6.963ms - LoadTime: 39.465us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 193.396ms - PrepareTime: 198.185ms AGGREGATION_NODE (id=6):(Total: 8s967ms, non-child: 200.279ms, % non-child: 2.23%) ExecOption: Codegen Enabled - BuildTime: 126.667us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 24.00 KB (24576) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 EXCHANGE_NODE (id=5):(Total: 8s767ms, non-child: 29.792us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 64.00 B (64) - ConvertRowBatchTime: 4.262us - DeserializeRowBatchTimer: 12.987us - FirstBatchArrivalWaitTime: 8s765ms - PeakMemoryUsage: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F01:(Total: 9s128ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:9s177ms max:9s180ms mean: 9s179ms stddev:1.398ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 4 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6609920) - PerHostPeakMemUsage: 653.11 MB (684837888) - PrepareTime: 195.558ms - RowsProduced: 1 - TotalCpuTime: 443.219ms - TotalNetworkReceiveTime: 8s931ms - TotalNetworkSendTime: 276.821us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 438.497ms, non-child: 438.497ms, % non-child: 100.00%) - CodegenTime: 2.256ms - CompileTime: 31.312ms - LoadTime: 77.531us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 214.235ms - PrepareTime: 191.539ms DataStreamSender (dst_id=5):(Total: 38.916us, non-child: 38.916us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 92.54 KB/sec - OverallThroughput: 417.06 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 10.483us - ThriftTransmitTime(*): 184.490us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s127ms, non-child: 193.310ms, % non-child: 2.12%) - BuildTime: 8.0us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s934ms, non-child: 2.759ms, % non-child: 0.03%) - BuildTime: 60.532us - GetNewBlockTime: 83.901us - GetResultsTime: 6.569us - HashBuckets: 5.12K (5120) - LargestPartitionPercent: 6 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6575104) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.172us EXCHANGE_NODE (id=3):(Total: 8s931ms, non-child: 8s931ms, % non-child: 100.00%) - BytesReceived: 220.00 B (220) - ConvertRowBatchTime: 12.432us - DeserializeRowBatchTimer: 32.624us - FirstBatchArrivalWaitTime: 7s061ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F00:(Total: 8s239ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 4.82 GB, max: 5.67 GB, avg: 5.44 GB, stddev: 364.68 MB completion times: min:7s101ms max:8s972ms mean: 8s314ms stddev:753.322ms execution rates: min:639.01 MB/sec max:701.37 MB/sec mean:671.63 MB/sec stddev:27.14 MB/sec num instances: 4 - AverageThreadTokens: 19.13 - PeakMemoryUsage: 646.83 MB (678252544) - PerHostPeakMemUsage: 653.11 MB (684837888) - PrepareTime: 202.924ms - RowsProduced: 20 - TotalCpuTime: 2m42s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.248ms - TotalStorageWaitTime: 1s651ms CodeGen:(Total: 475.56ms, non-child: 475.56ms, % non-child: 100.00%) - CodegenTime: 2.674ms - CompileTime: 42.184ms - LoadTime: 77.526us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 233.250ms - PrepareTime: 198.209ms DataStreamSender (dst_id=3):(Total: 37.849us, non-child: 37.849us, % non-child: 100.00%) - BytesSent: 220.00 B (220) - NetworkThroughput(*): 220.26 KB/sec - OverallThroughput: 5.54 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 37.316us - ThriftTransmitTime(*): 981.876us - UncompressedRowBatchSize: 256.00 B (256) AGGREGATION_NODE (id=1):(Total: 8s238ms, non-child: 6s918ms, % non-child: 83.98%) - BuildTime: 6s496ms - GetNewBlockTime: 115.761us - GetResultsTime: 23.916us - HashBuckets: 20.48K (20480) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.34us HDFS_SCAN_NODE (id=0):(Total: 1s319ms, non-child: 1s319ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.53 - AverageScannerThreadConcurrency: 19.27 - BytesRead: 5.44 GB (5838667196) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.44 GB (5838661564) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.44 GB (5838661564) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 21 - PeakMemoryUsage: 640.50 MB (671612928) - PerReadThreadRawHdfsThroughput: 1.55 GB/sec - RemoteScanRanges: 16 - RowsRead: 250.00M (250000000) - RowsReturned: 250.00M (250000000) - RowsReturnedRate: 191.28 M/sec - ScanRangesComplete: 67 - ScannerThreadsInvoluntaryContextSwitches: 8.44K (8444) - ScannerThreadsTotalWallClockTime: 2m35s - DelimiterParseTime: 13s651ms - MaterializeTupleTime(*): 7s802ms - ScannerThreadsSysTime: 1s179ms - ScannerThreadsUserTime: 25s212ms - ScannerThreadsVoluntaryContextSwitches: 268.23K (268231) - TotalRawHdfsReadTime(*): 3s504ms - TotalReadThroughput: 661.61 MB/sec Fragment F01: Instance 8d42458318c7ceed:a414fa6edfa8a9e (host=a1732.halxg.cloudera.com:22000):(Total: 9s132ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 710.66 MB (745185280) - PrepareTime: 198.265ms - RowsProduced: 1 - TotalCpuTime: 442.235ms - TotalNetworkReceiveTime: 8s933ms - TotalNetworkSendTime: 315.328us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 437.292ms, non-child: 437.292ms, % non-child: 100.00%) - CodegenTime: 2.290ms - CompileTime: 30.453ms - LoadTime: 85.626us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 211.236ms - PrepareTime: 194.51ms DataStreamSender (dst_id=5):(Total: 53.326us, non-child: 53.326us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 74.64 KB/sec - OverallThroughput: 293.01 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 26.587us - ThriftTransmitTime(*): 209.331us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s131ms, non-child: 195.970ms, % non-child: 2.15%) ExecOption: Codegen Enabled - BuildTime: 7.842us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s935ms, non-child: 2.776ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 113.764us - GetNewBlockTime: 90.53us - GetResultsTime: 4.377us - HashBuckets: 4.10K (4096) - LargestPartitionPercent: 6 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6574080) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.247us EXCHANGE_NODE (id=3):(Total: 8s933ms, non-child: 45.243us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 48.00 B, 48.00 B, 96.00 B - BytesReceived: 192.00 B (192) - ConvertRowBatchTime: 7.963us - DeserializeRowBatchTimer: 37.194us - FirstBatchArrivalWaitTime: 7s063ms - PeakMemoryUsage: 0 - RowsReturned: 16 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 8d42458318c7ceed:a414fa6edfa8a9c (host=a1731.halxg.cloudera.com:22000):(Total: 9s129ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 742.66 MB (778739712) - PrepareTime: 194.278ms - RowsProduced: 1 - TotalCpuTime: 440.433ms - TotalNetworkReceiveTime: 8s933ms - TotalNetworkSendTime: 303.505us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 435.907ms, non-child: 435.907ms, % non-child: 100.00%) - CodegenTime: 2.251ms - CompileTime: 30.977ms - LoadTime: 71.214us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 213.233ms - PrepareTime: 190.243ms DataStreamSender (dst_id=5):(Total: 34.301us, non-child: 34.301us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 72.87 KB/sec - OverallThroughput: 455.53 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.628us - ThriftTransmitTime(*): 214.417us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s128ms, non-child: 192.47ms, % non-child: 2.10%) ExecOption: Codegen Enabled - BuildTime: 2.545us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s936ms, non-child: 2.722ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 7.803us - GetNewBlockTime: 85.211us - GetResultsTime: 4.313us - HashBuckets: 2.05K (2048) - LargestPartitionPercent: 12 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 2 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.40us EXCHANGE_NODE (id=3):(Total: 8s933ms, non-child: 49.511us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 28.00 B, 28.00 B, 28.00 B, 56.00 B - BytesReceived: 112.00 B (112) - ConvertRowBatchTime: 7.298us - DeserializeRowBatchTimer: 21.87us - FirstBatchArrivalWaitTime: 7s063ms - PeakMemoryUsage: 0 - RowsReturned: 8 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 8d42458318c7ceed:a414fa6edfa8a9d (host=a1733.halxg.cloudera.com:22000):(Total: 9s128ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6606848) - PerHostPeakMemUsage: 664.66 MB (696950784) - PrepareTime: 194.463ms - RowsProduced: 1 - TotalCpuTime: 442.135ms - TotalNetworkReceiveTime: 8s932ms - TotalNetworkSendTime: 325.506us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 437.194ms, non-child: 437.194ms, % non-child: 100.00%) - CodegenTime: 2.236ms - CompileTime: 30.945ms - LoadTime: 74.354us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 214.375ms - PrepareTime: 190.561ms DataStreamSender (dst_id=5):(Total: 35.190us, non-child: 35.190us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 74.98 KB/sec - OverallThroughput: 444.02 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 7.730us - ThriftTransmitTime(*): 208.391us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s127ms, non-child: 192.238ms, % non-child: 2.11%) ExecOption: Codegen Enabled - BuildTime: 9.948us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s935ms, non-child: 2.738ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 102.257us - GetNewBlockTime: 84.629us - GetResultsTime: 5.957us - HashBuckets: 5.12K (5120) - LargestPartitionPercent: 5 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6569984) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.197us EXCHANGE_NODE (id=3):(Total: 8s932ms, non-child: 65.11us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 56.00 B, 56.00 B, 112.00 B - BytesReceived: 224.00 B (224) - ConvertRowBatchTime: 22.850us - DeserializeRowBatchTimer: 37.490us - FirstBatchArrivalWaitTime: 7s062ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 8d42458318c7ceed:a414fa6edfa8a9f (host=a1730.halxg.cloudera.com:22000):(Total: 9s124ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 494.46 MB (518475776) - PrepareTime: 195.223ms - RowsProduced: 1 - TotalCpuTime: 448.73ms - TotalNetworkReceiveTime: 8s927ms - TotalNetworkSendTime: 162.948us - TotalStorageWaitTime: 0ns CodeGen:(Total: 443.596ms, non-child: 443.596ms, % non-child: 100.00%) - CodegenTime: 2.247ms - CompileTime: 32.874ms - LoadTime: 78.930us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 218.97ms - PrepareTime: 191.299ms DataStreamSender (dst_id=5):(Total: 32.847us, non-child: 32.847us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 147.65 KB/sec - OverallThroughput: 475.69 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.990us - ThriftTransmitTime(*): 105.822us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s123ms, non-child: 192.984ms, % non-child: 2.12%) ExecOption: Codegen Enabled - BuildTime: 11.666us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s930ms, non-child: 2.801ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 18.305us - GetNewBlockTime: 75.713us - GetResultsTime: 11.630us - HashBuckets: 9.22K (9216) - LargestPartitionPercent: 2 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 9 - RowsReturnedRate: 1.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.204us EXCHANGE_NODE (id=3):(Total: 8s927ms, non-child: 161.963us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 88.00 B, 88.00 B, 88.00 B, 176.00 B - BytesReceived: 352.00 B (352) - ConvertRowBatchTime: 11.619us - DeserializeRowBatchTimer: 34.726us - FirstBatchArrivalWaitTime: 7s057ms - PeakMemoryUsage: 0 - RowsReturned: 36 - RowsReturnedRate: 4.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Fragment F00: Instance 8d42458318c7ceed:a414fa6edfa8aa3 (host=a1730.halxg.cloudera.com:22000):(Total: 8s885ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB MemoryUsage(500.0ms): 6.32 MB, 368.14 MB, 368.17 MB, 346.17 MB, 368.17 MB, 368.14 MB, 160.14 MB, 384.17 MB, 368.17 MB, 340.15 MB, 352.17 MB, 352.15 MB, 336.14 MB, 368.17 MB, 344.17 MB, 344.14 MB, 296.06 MB, 87.88 MB ThreadUsage(500.0ms): 1, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 13, 6 - AverageThreadTokens: 14.44 - PeakMemoryUsage: 488.17 MB (511881216) - PerHostPeakMemUsage: 494.46 MB (518475776) - PrepareTime: 202.306ms - RowsProduced: 20 - TotalCpuTime: 2m12s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.398ms - TotalStorageWaitTime: 943.558ms CodeGen:(Total: 484.642ms, non-child: 484.642ms, % non-child: 100.00%) - CodegenTime: 2.728ms - CompileTime: 44.116ms - LoadTime: 92.242us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 241.570ms - PrepareTime: 197.516ms DataStreamSender (dst_id=3):(Total: 37.295us, non-child: 37.295us, % non-child: 100.00%) - BytesSent: 220.00 B (220) - NetworkThroughput(*): 193.22 KB/sec - OverallThroughput: 5.63 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 45.507us - ThriftTransmitTime(*): 1.111ms - UncompressedRowBatchSize: 256.00 B (256) AGGREGATION_NODE (id=1):(Total: 8s884ms, non-child: 7s347ms, % non-child: 82.71%) ExecOption: Codegen Enabled - BuildTime: 6s906ms - GetNewBlockTime: 149.940us - GetResultsTime: 24.809us - HashBuckets: 20.48K (20480) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.84us HDFS_SCAN_NODE (id=0):(Total: 1s536ms, non-child: 1s536ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB Hdfs Read Thread Concurrency Bucket: 0:70.59% 1:23.53% 2:5.882% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:59 ExecOption: Codegen enabled: 59 out of 59 BytesRead(500.0ms): 0, 577.50 MB, 905.50 MB, 1.17 GB, 1.50 GB, 1.85 GB, 1.88 GB, 2.42 GB, 2.73 GB, 3.03 GB, 3.37 GB, 3.72 GB, 4.06 GB, 4.42 GB, 4.72 GB, 5.07 GB, 5.38 GB, 5.53 GB - AverageHdfsReadThreadConcurrency: 0.35 - AverageScannerThreadConcurrency: 14.24 - BytesRead: 5.60 GB (6012072860) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.60 GB (6012070812) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.60 GB (6012070812) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 15 - PeakMemoryUsage: 481.84 MB (505245696) - PerReadThreadRawHdfsThroughput: 1.57 GB/sec - RemoteScanRanges: 6 - RowsRead: 257.43M (257426663) - RowsReturned: 257.43M (257426663) - RowsReturnedRate: 167.54 M/sec - ScanRangesComplete: 59 - ScannerThreadsInvoluntaryContextSwitches: 10.06K (10062) - ScannerThreadsTotalWallClockTime: 2m4s - DelimiterParseTime: 13s994ms - MaterializeTupleTime(*): 7s940ms - ScannerThreadsSysTime: 1s283ms - ScannerThreadsUserTime: 25s968ms - ScannerThreadsVoluntaryContextSwitches: 278.96K (278963) - TotalRawHdfsReadTime(*): 3s559ms - TotalReadThroughput: 629.06 MB/sec Instance 8d42458318c7ceed:a414fa6edfa8aa0 (host=a1731.halxg.cloudera.com:22000):(Total: 8s842ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB MemoryUsage(500.0ms): 6.32 MB, 552.36 MB, 568.39 MB, 560.36 MB, 560.39 MB, 552.36 MB, 568.39 MB, 528.36 MB, 440.36 MB, 492.39 MB, 568.36 MB, 544.36 MB, 556.39 MB, 568.39 MB, 528.36 MB, 488.33 MB, 368.25 MB, 62.43 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 22, 20, 4 - AverageThreadTokens: 21.28 - PeakMemoryUsage: 736.39 MB (772157440) - PerHostPeakMemUsage: 742.66 MB (778739712) - PrepareTime: 201.888ms - RowsProduced: 20 - TotalCpuTime: 3m15s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.142ms - TotalStorageWaitTime: 1s228ms CodeGen:(Total: 473.592ms, non-child: 473.592ms, % non-child: 100.00%) - CodegenTime: 2.653ms - CompileTime: 41.430ms - LoadTime: 71.294us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 233.471ms - PrepareTime: 197.303ms DataStreamSender (dst_id=3):(Total: 37.754us, non-child: 37.754us, % non-child: 100.00%) - BytesSent: 220.00 B (220) - NetworkThroughput(*): 241.93 KB/sec - OverallThroughput: 5.56 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 35.886us - ThriftTransmitTime(*): 888.51us - UncompressedRowBatchSize: 256.00 B (256) AGGREGATION_NODE (id=1):(Total: 8s841ms, non-child: 7s562ms, % non-child: 85.53%) ExecOption: Codegen Enabled - BuildTime: 7s135ms - GetNewBlockTime: 111.761us - GetResultsTime: 25.134us - HashBuckets: 20.48K (20480) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.12us HDFS_SCAN_NODE (id=0):(Total: 1s278ms, non-child: 1s278ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB Hdfs Read Thread Concurrency Bucket: 0:58.82% 1:35.29% 2:0% 3:0% 4:5.882% 5:0% 6:0% File Formats: TEXT/NONE:53 ExecOption: Codegen enabled: 53 out of 53 BytesRead(500.0ms): 0, 728.21 MB, 1.08 GB, 1.45 GB, 1.77 GB, 2.09 GB, 2.45 GB, 2.73 GB, 2.96 GB, 3.27 GB, 3.69 GB, 4.00 GB, 4.32 GB, 4.70 GB, 5.02 GB, 5.33 GB, 5.57 GB, 5.64 GB - AverageHdfsReadThreadConcurrency: 0.59 - AverageScannerThreadConcurrency: 21.47 - BytesRead: 5.67 GB (6085373526) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.67 GB (6085367382) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.67 GB (6085367382) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 730.06 MB (765521920) - PerReadThreadRawHdfsThroughput: 1.56 GB/sec - RemoteScanRanges: 18 - RowsRead: 260.57M (260565882) - RowsReturned: 260.57M (260565882) - RowsReturnedRate: 203.74 M/sec - ScanRangesComplete: 53 - ScannerThreadsInvoluntaryContextSwitches: 8.34K (8343) - ScannerThreadsTotalWallClockTime: 3m7s - DelimiterParseTime: 14s254ms - MaterializeTupleTime(*): 8s104ms - ScannerThreadsSysTime: 1s198ms - ScannerThreadsUserTime: 26s344ms - ScannerThreadsVoluntaryContextSwitches: 275.87K (275874) - TotalRawHdfsReadTime(*): 3s635ms - TotalReadThroughput: 641.27 MB/sec Instance 8d42458318c7ceed:a414fa6edfa8aa2 (host=a1732.halxg.cloudera.com:22000):(Total: 8s198ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB MemoryUsage(500.0ms): 0, 696.39 MB, 544.36 MB, 560.37 MB, 560.36 MB, 560.39 MB, 560.39 MB, 560.39 MB, 384.39 MB, 540.34 MB, 568.39 MB, 568.39 MB, 530.39 MB, 512.32 MB, 504.33 MB, 444.30 MB, 192.11 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 22, 22, 21, 14 - AverageThreadTokens: 21.65 - PeakMemoryUsage: 704.39 MB (738603008) - PerHostPeakMemUsage: 710.66 MB (745185280) - PrepareTime: 205.350ms - RowsProduced: 20 - TotalCpuTime: 3m - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.140ms - TotalStorageWaitTime: 1s475ms CodeGen:(Total: 472.565ms, non-child: 472.565ms, % non-child: 100.00%) - CodegenTime: 2.688ms - CompileTime: 41.787ms - LoadTime: 82.617us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 228.726ms - PrepareTime: 200.546ms DataStreamSender (dst_id=3):(Total: 38.657us, non-child: 38.657us, % non-child: 100.00%) - BytesSent: 220.00 B (220) - NetworkThroughput(*): 225.12 KB/sec - OverallThroughput: 5.43 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 31.462us - ThriftTransmitTime(*): 954.336us - UncompressedRowBatchSize: 256.00 B (256) AGGREGATION_NODE (id=1):(Total: 8s197ms, non-child: 6s810ms, % non-child: 83.08%) ExecOption: Codegen Enabled - BuildTime: 6s384ms - GetNewBlockTime: 103.715us - GetResultsTime: 21.625us - HashBuckets: 20.48K (20480) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 1.939us HDFS_SCAN_NODE (id=0):(Total: 1s387ms, non-child: 1s387ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB Hdfs Read Thread Concurrency Bucket: 0:68.75% 1:25% 2:6.25% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:64 ExecOption: Codegen enabled: 64 out of 64 BytesRead(500.0ms): 697.70 MB, 921.70 MB, 1.26 GB, 1.60 GB, 1.97 GB, 2.33 GB, 2.70 GB, 2.88 GB, 3.36 GB, 3.72 GB, 4.11 GB, 4.44 GB, 4.78 GB, 5.13 GB, 5.45 GB, 5.61 GB - AverageHdfsReadThreadConcurrency: 0.38 - AverageScannerThreadConcurrency: 21.94 - BytesRead: 5.66 GB (6079010449) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.66 GB (6079009425) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.66 GB (6079009425) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 698.04 MB (731951104) - PerReadThreadRawHdfsThroughput: 1.50 GB/sec - RemoteScanRanges: 3 - RowsRead: 260.29M (260289143) - RowsReturned: 260.29M (260289143) - RowsReturnedRate: 187.66 M/sec - ScanRangesComplete: 64 - ScannerThreadsInvoluntaryContextSwitches: 8.61K (8607) - ScannerThreadsTotalWallClockTime: 2m53s - DelimiterParseTime: 14s587ms - MaterializeTupleTime(*): 8s349ms - ScannerThreadsSysTime: 1s233ms - ScannerThreadsUserTime: 26s753ms - ScannerThreadsVoluntaryContextSwitches: 279.58K (279577) - TotalRawHdfsReadTime(*): 3s779ms - TotalReadThroughput: 717.67 MB/sec Instance 8d42458318c7ceed:a414fa6edfa8aa1 (host=a1733.halxg.cloudera.com:22000):(Total: 7s031ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB MemoryUsage(500.0ms): 6.32 MB, 554.39 MB, 576.36 MB, 528.39 MB, 560.36 MB, 560.37 MB, 540.39 MB, 540.39 MB, 424.39 MB, 466.36 MB, 416.22 MB, 368.17 MB, 368.17 MB, 336.14 MB, 70.76 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 18, 16, 16, 16, 4 - AverageThreadTokens: 19.13 - PeakMemoryUsage: 658.39 MB (690368512) - PerHostPeakMemUsage: 664.66 MB (696950784) - PrepareTime: 202.153ms - RowsProduced: 20 - TotalCpuTime: 2m21s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.310ms - TotalStorageWaitTime: 2s960ms CodeGen:(Total: 469.424ms, non-child: 469.424ms, % non-child: 100.00%) - CodegenTime: 2.625ms - CompileTime: 41.403ms - LoadTime: 63.951us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 229.233ms - PrepareTime: 197.468ms DataStreamSender (dst_id=3):(Total: 37.691us, non-child: 37.691us, % non-child: 100.00%) - BytesSent: 220.00 B (220) - NetworkThroughput(*): 220.76 KB/sec - OverallThroughput: 5.57 MB/sec - PeakMemoryUsage: 192.00 KB (196608) - SerializeBatchTime: 36.410us - ThriftTransmitTime(*): 973.188us - UncompressedRowBatchSize: 256.00 B (256) AGGREGATION_NODE (id=1):(Total: 7s029ms, non-child: 5s954ms, % non-child: 84.70%) ExecOption: Codegen Enabled - BuildTime: 5s557ms - GetNewBlockTime: 97.629us - GetResultsTime: 24.97us - HashBuckets: 20.48K (20480) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.29 MB (6590464) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.102us HDFS_SCAN_NODE (id=0):(Total: 1s075ms, non-child: 1s075ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB Hdfs Read Thread Concurrency Bucket: 0:57.14% 1:21.43% 2:7.143% 3:14.29% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:94 ExecOption: Codegen enabled: 94 out of 94 BytesRead(500.0ms): 0, 595.29 MB, 957.05 MB, 1.24 GB, 1.64 GB, 2.01 GB, 2.33 GB, 2.69 GB, 2.94 GB, 3.35 GB, 3.64 GB, 4.00 GB, 4.39 GB, 4.70 GB, 4.82 GB - AverageHdfsReadThreadConcurrency: 0.79 - AverageScannerThreadConcurrency: 19.43 - BytesRead: 4.82 GB (5178211951) - BytesReadDataNodeCache: 0 - BytesReadLocal: 4.82 GB (5178198639) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 4.82 GB (5178198639) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 652.06 MB (683732992) - PerReadThreadRawHdfsThroughput: 1.59 GB/sec - RemoteScanRanges: 39 - RowsRead: 221.72M (221718312) - RowsReturned: 221.72M (221718312) - RowsReturnedRate: 206.20 M/sec - ScanRangesComplete: 94 - ScannerThreadsInvoluntaryContextSwitches: 6.77K (6767) - ScannerThreadsTotalWallClockTime: 2m16s - DelimiterParseTime: 11s771ms - MaterializeTupleTime(*): 6s813ms - ScannerThreadsSysTime: 999.836ms - ScannerThreadsUserTime: 21s783ms - ScannerThreadsVoluntaryContextSwitches: 238.51K (238511) - TotalRawHdfsReadTime(*): 3s041ms - TotalReadThroughput: 658.44 MB/sec [localhost:21000] > alter table billion_rows_text change x x bigint; [localhost:21000] > alter table billion_rows_text change y y bigint; [localhost:21000] > alter table billion_rows_text change z z bigint; [localhost:21000] > select count(distinct x) from billion_rows_text; +-------------------+ | count(distinct x) | +-------------------+ | 18 | +-------------------+ [localhost:21000] > profile; Query Runtime Profile: Query (id=634de4e3144b73be:966fad5b7fccc90): Summary: Session ID: 8f4cad0b2b385c70:b41743dae24d22ad Session Type: BEESWAX Start Time: 2015-01-12 15:15:17.719365000 End Time: 2015-01-12 15:15:27.202675000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) User: jrussell Connected User: jrussell Delegated User: Network Address: 127.0.0.1:38627 Default Db: oreilly Sql Statement: select count(distinct x) from billion_rows_text Coordinator: a1730.halxg.cloudera.com:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=3.88GB VCores=2 WARNING: The following tables are missing relevant table and/or column statistics. oreilly.billion_rows_text F02:PLAN FRAGMENT [UNPARTITIONED] 06:AGGREGATE [FINALIZE] | output: count:merge(x) | hosts=4 per-host-mem=unavailable | tuple-ids=2 row-size=8B cardinality=1 | 05:EXCHANGE [UNPARTITIONED] hosts=4 per-host-mem=unavailable tuple-ids=2 row-size=8B cardinality=1 F01:PLAN FRAGMENT [HASH(x)] DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=05, UNPARTITIONED] 02:AGGREGATE | output: count(x) | hosts=4 per-host-mem=10.00MB | tuple-ids=2 row-size=8B cardinality=1 | 04:AGGREGATE | group by: x | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=8B cardinality=unavailable | 03:EXCHANGE [HASH(x)] hosts=4 per-host-mem=0B tuple-ids=1 row-size=8B cardinality=unavailable F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=03, HASH(x)] 01:AGGREGATE | group by: x | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=8B cardinality=unavailable | 00:SCAN HDFS [oreilly.billion_rows_text, RANDOM] partitions=1/1 files=113 size=21.75GB table stats: unavailable column stats: unavailable hosts=4 per-host-mem=3.75GB tuple-ids=0 row-size=8B cardinality=unavailable ---------------- Estimated Per-Host Mem: 4160749568 Estimated Per-Host VCores: 2 Tables Missing Stats: oreilly.billion_rows_text Admission result: Admitted immediately Request Pool: root.jrussell ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail -------------------------------------------------------------------------------------------------------------------- 06:AGGREGATE 1 195.826ms 195.826ms 1 1 28.00 KB -1.00 B FINALIZE 05:EXCHANGE 1 50.726us 50.726us 4 1 0 -1.00 B UNPARTITIONED 02:AGGREGATE 4 194.103ms 195.512ms 4 1 12.00 KB 10.00 MB 04:AGGREGATE 4 2.755ms 2.850ms 18 -1 6.27 MB 128.00 MB 03:EXCHANGE 4 89.201us 150.338us 72 -1 0 0 HASH(x) 01:AGGREGATE 4 6s929ms 7s346ms 72 -1 6.31 MB 128.00 MB 00:SCAN HDFS 4 1s338ms 1s432ms 1.00B -1 732.07 MB 3.75 GB oreilly.billion_rows_text Query Timeline: 9s487ms - Start execution: 60.818us (60.818us) - Planning finished: 26.458ms (26.397ms) - Submit for admission: 27.63ms (605.101us) - Completed admission: 27.143ms (80.85us) - Ready to start remote fragments: 223.994ms (196.851ms) - Remote fragments started: 633.391ms (409.396ms) - Rows available: 9s470ms (8s837ms) - First row fetched: 9s480ms (10.139ms) - Unregister query: 9s483ms (2.402ms) ImpalaServer: - ClientFetchWaitTimer: 11.446ms - RowMaterializationTimer: 5.938us Execution Profile 634de4e3144b73be:966fad5b7fccc90:(Total: 9s443ms, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 8, last: 0.208569, min: 0.196051, max: 0.208569, mean: 0.201582, stddev: 0.00447392 Per Node Peak Memory Usage: a1733.halxg.cloudera.com:22000(744.66 MB) a1731.halxg.cloudera.com:22000(714.63 MB) a1732.halxg.cloudera.com:22000(722.63 MB) a1730.halxg.cloudera.com:22000(478.43 MB) - FinalizationTimer: 0ns Coordinator Fragment F02:(Total: 8s835ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB ThreadUsage(500.0ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 40.00 KB (40960) - PerHostPeakMemUsage: 0 - PrepareTime: 195.580ms - RowsProduced: 1 - TotalCpuTime: 813.95ms - TotalNetworkReceiveTime: 8s639ms - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 392.134ms, non-child: 392.134ms, % non-child: 100.00%) - CodegenTime: 479.135us - CompileTime: 6.811ms - LoadTime: 37.948us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 190.198ms - PrepareTime: 193.810ms AGGREGATION_NODE (id=6):(Total: 8s835ms, non-child: 195.826ms, % non-child: 2.22%) ExecOption: Codegen Enabled - BuildTime: 2.570us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 28.00 KB (28672) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 EXCHANGE_NODE (id=5):(Total: 8s639ms, non-child: 50.726us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 64.00 B (64) - ConvertRowBatchTime: 6.614us - DeserializeRowBatchTimer: 19.692us - FirstBatchArrivalWaitTime: 8s638ms - PeakMemoryUsage: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F01:(Total: 8s999ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:9s047ms max:9s050ms mean: 9s048ms stddev:846.157us execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 4 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6607872) - PerHostPeakMemUsage: 665.09 MB (697396224) - PrepareTime: 196.376ms - RowsProduced: 1 - TotalCpuTime: 443.69ms - TotalNetworkReceiveTime: 8s801ms - TotalNetworkSendTime: 286.912us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 438.495ms, non-child: 438.495ms, % non-child: 100.00%) - CodegenTime: 2.257ms - CompileTime: 30.881ms - LoadTime: 80.596us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 213.873ms - PrepareTime: 192.353ms DataStreamSender (dst_id=5):(Total: 32.974us, non-child: 32.974us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 91.85 KB/sec - OverallThroughput: 475.90 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 5.253us - ThriftTransmitTime(*): 186.461us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 8s998ms, non-child: 194.103ms, % non-child: 2.16%) - BuildTime: 7.687us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s804ms, non-child: 2.755ms, % non-child: 0.03%) - BuildTime: 32.210us - GetNewBlockTime: 93.393us - GetResultsTime: 7.510us - HashBuckets: 4.61K (4608) - LargestPartitionPercent: 7 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6572032) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.85us EXCHANGE_NODE (id=3):(Total: 8s801ms, non-child: 8s801ms, % non-child: 100.00%) - BytesReceived: 256.00 B (256) - ConvertRowBatchTime: 8.394us - DeserializeRowBatchTimer: 30.533us - FirstBatchArrivalWaitTime: 7s215ms - PeakMemoryUsage: 0 - RowsReturned: 18 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F00:(Total: 8s269ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 4.82 GB, max: 5.67 GB, avg: 5.44 GB, stddev: 364.68 MB completion times: min:7s252ms max:8s840ms mean: 8s342ms stddev:634.691ms execution rates: min:648.58 MB/sec max:680.87 MB/sec mean:668.07 MB/sec stddev:11.96 MB/sec num instances: 4 - AverageThreadTokens: 20.02 - PeakMemoryUsage: 658.81 MB (690810880) - PerHostPeakMemUsage: 665.09 MB (697396224) - PrepareTime: 203.558ms - RowsProduced: 18 - TotalCpuTime: 2m44s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.148ms - TotalStorageWaitTime: 1s336ms CodeGen:(Total: 474.942ms, non-child: 474.942ms, % non-child: 100.00%) - CodegenTime: 2.671ms - CompileTime: 41.868ms - LoadTime: 77.141us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 232.823ms - PrepareTime: 198.792ms DataStreamSender (dst_id=3):(Total: 42.21us, non-child: 42.21us, % non-child: 100.00%) - BytesSent: 256.00 B (256) - NetworkThroughput(*): 276.26 KB/sec - OverallThroughput: 6.21 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 38.842us - ThriftTransmitTime(*): 912.22us - UncompressedRowBatchSize: 376.00 B (376) AGGREGATION_NODE (id=1):(Total: 8s268ms, non-child: 6s929ms, % non-child: 83.81%) - BuildTime: 6s499ms - GetNewBlockTime: 113.118us - GetResultsTime: 24.556us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.0us HDFS_SCAN_NODE (id=0):(Total: 1s338ms, non-child: 1s338ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.54 - AverageScannerThreadConcurrency: 19.85 - BytesRead: 5.44 GB (5838667196) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.44 GB (5838661564) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.44 GB (5838661564) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 21 - PeakMemoryUsage: 652.51 MB (684201984) - PerReadThreadRawHdfsThroughput: 1.52 GB/sec - RemoteScanRanges: 22 - RowsRead: 250.00M (250000000) - RowsReturned: 250.00M (250000000) - RowsReturnedRate: 187.49 M/sec - ScanRangesComplete: 67 - ScannerThreadsInvoluntaryContextSwitches: 9.31K (9308) - ScannerThreadsTotalWallClockTime: 2m37s - DelimiterParseTime: 13s848ms - MaterializeTupleTime(*): 10s257ms - ScannerThreadsSysTime: 1s291ms - ScannerThreadsUserTime: 28s114ms - ScannerThreadsVoluntaryContextSwitches: 272.94K (272945) - TotalRawHdfsReadTime(*): 3s585ms - TotalReadThroughput: 661.25 MB/sec Fragment F01: Instance 634de4e3144b73be:966fad5b7fccc94 (host=a1732.halxg.cloudera.com:22000):(Total: 9s000ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.29 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6606848) - PerHostPeakMemUsage: 722.63 MB (757735424) - PrepareTime: 197.873ms - RowsProduced: 1 - TotalCpuTime: 443.794ms - TotalNetworkReceiveTime: 8s801ms - TotalNetworkSendTime: 359.27us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 438.918ms, non-child: 438.918ms, % non-child: 100.00%) - CodegenTime: 2.324ms - CompileTime: 30.394ms - LoadTime: 81.194us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 213.373ms - PrepareTime: 193.636ms DataStreamSender (dst_id=5):(Total: 32.319us, non-child: 32.319us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 64.29 KB/sec - OverallThroughput: 483.46 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 5.724us - ThriftTransmitTime(*): 243.27us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 8s999ms, non-child: 195.512ms, % non-child: 2.17%) ExecOption: Codegen Enabled - BuildTime: 7.115us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s804ms, non-child: 2.850ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 12.10us - GetNewBlockTime: 96.331us - GetResultsTime: 8.242us - HashBuckets: 4.10K (4096) - LargestPartitionPercent: 6 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6569984) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.190us EXCHANGE_NODE (id=3):(Total: 8s801ms, non-child: 48.516us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58.00 B, 58.00 B, 174.00 B - BytesReceived: 232.00 B (232) - ConvertRowBatchTime: 8.346us - DeserializeRowBatchTimer: 30.850us - FirstBatchArrivalWaitTime: 7s214ms - PeakMemoryUsage: 0 - RowsReturned: 16 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 634de4e3144b73be:966fad5b7fccc92 (host=a1731.halxg.cloudera.com:22000):(Total: 8s998ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.29 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6606848) - PerHostPeakMemUsage: 714.63 MB (749346816) - PrepareTime: 196.5ms - RowsProduced: 1 - TotalCpuTime: 442.421ms - TotalNetworkReceiveTime: 8s801ms - TotalNetworkSendTime: 380.209us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 437.806ms, non-child: 437.806ms, % non-child: 100.00%) - CodegenTime: 2.215ms - CompileTime: 30.797ms - LoadTime: 73.184us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 213.548ms - PrepareTime: 192.83ms DataStreamSender (dst_id=5):(Total: 29.769us, non-child: 29.769us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 74.20 KB/sec - OverallThroughput: 524.87 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.850us - ThriftTransmitTime(*): 210.577us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 8s998ms, non-child: 193.816ms, % non-child: 2.15%) ExecOption: Codegen Enabled - BuildTime: 8.669us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s804ms, non-child: 2.645ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 94.494us - GetNewBlockTime: 74.794us - GetResultsTime: 14.460us - HashBuckets: 9.22K (9216) - LargestPartitionPercent: 2 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6569984) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 9 - RowsReturnedRate: 1.00 /sec - SpilledPartitions: 0 - UnpinTime: 1.939us EXCHANGE_NODE (id=3):(Total: 8s801ms, non-child: 40.142us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 116.00 B, 116.00 B, 116.00 B - BytesReceived: 464.00 B (464) - ConvertRowBatchTime: 7.317us - DeserializeRowBatchTimer: 29.280us - FirstBatchArrivalWaitTime: 7s215ms - PeakMemoryUsage: 0 - RowsReturned: 36 - RowsReturnedRate: 4.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 634de4e3144b73be:966fad5b7fccc93 (host=a1733.halxg.cloudera.com:22000):(Total: 8s998ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.29 MB, 6.29 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6606848) - PerHostPeakMemUsage: 744.66 MB (780836864) - PrepareTime: 196.695ms - RowsProduced: 1 - TotalCpuTime: 443.731ms - TotalNetworkReceiveTime: 8s800ms - TotalNetworkSendTime: 260.39us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 439.238ms, non-child: 439.238ms, % non-child: 100.00%) - CodegenTime: 2.213ms - CompileTime: 31.108ms - LoadTime: 75.560us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 214.32ms - PrepareTime: 192.772ms DataStreamSender (dst_id=5):(Total: 34.780us, non-child: 34.780us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 85.11 KB/sec - OverallThroughput: 449.25 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 7.695us - ThriftTransmitTime(*): 183.591us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 8s998ms, non-child: 194.454ms, % non-child: 2.16%) ExecOption: Codegen Enabled - BuildTime: 9.316us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s803ms, non-child: 2.749ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 13.618us - GetNewBlockTime: 101.396us - GetResultsTime: 4.790us - HashBuckets: 3.07K (3072) - LargestPartitionPercent: 8 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6569984) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 3 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.220us EXCHANGE_NODE (id=3):(Total: 8s801ms, non-child: 117.810us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 47.00 B, 47.00 B, 47.00 B, 141.00 B - BytesReceived: 188.00 B (188) - ConvertRowBatchTime: 7.694us - DeserializeRowBatchTimer: 28.33us - FirstBatchArrivalWaitTime: 7s214ms - PeakMemoryUsage: 0 - RowsReturned: 12 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 634de4e3144b73be:966fad5b7fccc95 (host=a1730.halxg.cloudera.com:22000):(Total: 8s998ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.29 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 478.43 MB (501665792) - PrepareTime: 194.929ms - RowsProduced: 1 - TotalCpuTime: 442.332ms - TotalNetworkReceiveTime: 8s802ms - TotalNetworkSendTime: 148.376us - TotalStorageWaitTime: 0ns CodeGen:(Total: 438.17ms, non-child: 438.17ms, % non-child: 100.00%) - CodegenTime: 2.278ms - CompileTime: 31.226ms - LoadTime: 92.447us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 214.541ms - PrepareTime: 190.922ms DataStreamSender (dst_id=5):(Total: 35.31us, non-child: 35.31us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 143.81 KB/sec - OverallThroughput: 446.03 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.745us - ThriftTransmitTime(*): 108.652us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 8s998ms, non-child: 192.632ms, % non-child: 2.14%) ExecOption: Codegen Enabled - BuildTime: 5.651us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s805ms, non-child: 2.774ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 8.718us - GetNewBlockTime: 101.54us - GetResultsTime: 2.550us - HashBuckets: 2.05K (2048) - LargestPartitionPercent: 12 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 2 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 1.992us EXCHANGE_NODE (id=3):(Total: 8s802ms, non-child: 150.338us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 35.00 B, 35.00 B, 35.00 B - BytesReceived: 140.00 B (140) - ConvertRowBatchTime: 10.221us - DeserializeRowBatchTimer: 33.969us - FirstBatchArrivalWaitTime: 7s215ms - PeakMemoryUsage: 0 - RowsReturned: 8 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Fragment F00: Instance 634de4e3144b73be:966fad5b7fccc99 (host=a1730.halxg.cloudera.com:22000):(Total: 8s759ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB MemoryUsage(500.0ms): 6.29 MB, 392.14 MB, 360.14 MB, 352.10 MB, 368.14 MB, 376.11 MB, 216.09 MB, 376.14 MB, 346.11 MB, 376.11 MB, 360.14 MB, 368.11 MB, 240.11 MB, 384.14 MB, 368.14 MB, 348.09 MB, 296.03 MB, 119.95 MB ThreadUsage(500.0ms): 1, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 13, 9 - AverageThreadTokens: 14.56 - PeakMemoryUsage: 472.14 MB (495071232) - PerHostPeakMemUsage: 478.43 MB (501665792) - PrepareTime: 204.99ms - RowsProduced: 18 - TotalCpuTime: 2m9s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.149ms - TotalStorageWaitTime: 953.768ms CodeGen:(Total: 482.454ms, non-child: 482.454ms, % non-child: 100.00%) - CodegenTime: 2.817ms - CompileTime: 42.967ms - LoadTime: 68.639us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 238.872ms - PrepareTime: 199.177ms DataStreamSender (dst_id=3):(Total: 38.83us, non-child: 38.83us, % non-child: 100.00%) - BytesSent: 256.00 B (256) - NetworkThroughput(*): 287.25 KB/sec - OverallThroughput: 6.41 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 44.284us - ThriftTransmitTime(*): 870.311us - UncompressedRowBatchSize: 376.00 B (376) AGGREGATION_NODE (id=1):(Total: 8s758ms, non-child: 7s346ms, % non-child: 83.88%) ExecOption: Codegen Enabled - BuildTime: 6s902ms - GetNewBlockTime: 142.985us - GetResultsTime: 25.449us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 1.966us HDFS_SCAN_NODE (id=0):(Total: 1s411ms, non-child: 1s411ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB Hdfs Read Thread Concurrency Bucket: 0:64.71% 1:35.29% 2:0% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:59 ExecOption: Codegen enabled: 59 out of 59 BytesRead(500.0ms): 0, 461.79 MB, 738.68 MB, 1.07 GB, 1.40 GB, 1.74 GB, 1.93 GB, 2.39 GB, 2.70 GB, 3.06 GB, 3.35 GB, 3.68 GB, 3.89 GB, 4.35 GB, 4.67 GB, 5.04 GB, 5.33 GB, 5.54 GB - AverageHdfsReadThreadConcurrency: 0.35 - AverageScannerThreadConcurrency: 14.35 - BytesRead: 5.60 GB (6012072860) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.60 GB (6012070812) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.60 GB (6012070812) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 15 - PeakMemoryUsage: 465.84 MB (488468480) - PerReadThreadRawHdfsThroughput: 1.57 GB/sec - RemoteScanRanges: 8 - RowsRead: 257.43M (257426663) - RowsReturned: 257.43M (257426663) - RowsReturnedRate: 182.31 M/sec - ScanRangesComplete: 59 - ScannerThreadsInvoluntaryContextSwitches: 10.96K (10958) - ScannerThreadsTotalWallClockTime: 2m1s - DelimiterParseTime: 14s101ms - MaterializeTupleTime(*): 10s312ms - ScannerThreadsSysTime: 1s373ms - ScannerThreadsUserTime: 28s640ms - ScannerThreadsVoluntaryContextSwitches: 282.69K (282689) - TotalRawHdfsReadTime(*): 3s574ms - TotalReadThroughput: 629.95 MB/sec Instance 634de4e3144b73be:966fad5b7fccc98 (host=a1732.halxg.cloudera.com:22000):(Total: 8s588ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB MemoryUsage(500.0ms): 6.29 MB, 532.33 MB, 568.36 MB, 554.36 MB, 560.33 MB, 568.36 MB, 560.33 MB, 560.36 MB, 400.33 MB, 544.36 MB, 568.33 MB, 560.33 MB, 544.33 MB, 544.31 MB, 512.28 MB, 476.23 MB, 280.23 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 22, 21, 20 - AverageThreadTokens: 22.12 - PeakMemoryUsage: 716.36 MB (751153152) - PerHostPeakMemUsage: 722.63 MB (757735424) - PrepareTime: 206.373ms - RowsProduced: 18 - TotalCpuTime: 3m11s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.88ms - TotalStorageWaitTime: 1s514ms CodeGen:(Total: 476.576ms, non-child: 476.576ms, % non-child: 100.00%) - CodegenTime: 2.641ms - CompileTime: 41.605ms - LoadTime: 102.948us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 231.880ms - PrepareTime: 201.573ms DataStreamSender (dst_id=3):(Total: 39.197us, non-child: 39.197us, % non-child: 100.00%) - BytesSent: 256.00 B (256) - NetworkThroughput(*): 266.79 KB/sec - OverallThroughput: 6.23 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 35.873us - ThriftTransmitTime(*): 937.64us - UncompressedRowBatchSize: 376.00 B (376) AGGREGATION_NODE (id=1):(Total: 8s587ms, non-child: 7s154ms, % non-child: 83.31%) ExecOption: Codegen Enabled - BuildTime: 6s709ms - GetNewBlockTime: 129.210us - GetResultsTime: 25.226us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.38us HDFS_SCAN_NODE (id=0):(Total: 1s432ms, non-child: 1s432ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB Hdfs Read Thread Concurrency Bucket: 0:68.75% 1:25% 2:0% 3:6.25% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:64 ExecOption: Codegen enabled: 64 out of 64 BytesRead(500.0ms): 0, 709.58 MB, 1.06 GB, 1.39 GB, 1.72 GB, 2.05 GB, 2.39 GB, 2.74 GB, 2.91 GB, 3.38 GB, 3.75 GB, 4.11 GB, 4.43 GB, 4.80 GB, 5.14 GB, 5.44 GB, 5.62 GB - AverageHdfsReadThreadConcurrency: 0.44 - AverageScannerThreadConcurrency: 22.44 - BytesRead: 5.66 GB (6079010449) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.66 GB (6079009425) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.66 GB (6079009425) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 710.06 MB (744550400) - PerReadThreadRawHdfsThroughput: 1.50 GB/sec - RemoteScanRanges: 4 - RowsRead: 260.29M (260289143) - RowsReturned: 260.29M (260289143) - RowsReturnedRate: 181.66 M/sec - ScanRangesComplete: 64 - ScannerThreadsInvoluntaryContextSwitches: 9.29K (9288) - ScannerThreadsTotalWallClockTime: 3m3s - DelimiterParseTime: 14s441ms - MaterializeTupleTime(*): 10s763ms - ScannerThreadsSysTime: 1s271ms - ScannerThreadsUserTime: 29s318ms - ScannerThreadsVoluntaryContextSwitches: 283.72K (283716) - TotalRawHdfsReadTime(*): 3s769ms - TotalReadThroughput: 677.34 MB/sec Instance 634de4e3144b73be:966fad5b7fccc96 (host=a1731.halxg.cloudera.com:22000):(Total: 8s543ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB MemoryUsage(500.0ms): 6.29 MB, 676.33 MB, 568.33 MB, 560.36 MB, 548.33 MB, 568.33 MB, 544.36 MB, 544.36 MB, 448.36 MB, 488.33 MB, 544.34 MB, 568.34 MB, 560.36 MB, 560.33 MB, 552.33 MB, 512.33 MB, 408.21 MB, 95.95 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 23, 20, 9 - AverageThreadTokens: 21.61 - PeakMemoryUsage: 708.36 MB (742764544) - PerHostPeakMemUsage: 714.63 MB (749346816) - PrepareTime: 199.374ms - RowsProduced: 18 - TotalCpuTime: 3m12s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.282ms - TotalStorageWaitTime: 1s406ms CodeGen:(Total: 469.855ms, non-child: 469.855ms, % non-child: 100.00%) - CodegenTime: 2.560ms - CompileTime: 41.587ms - LoadTime: 64.428us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 231.962ms - PrepareTime: 194.856ms DataStreamSender (dst_id=3):(Total: 29.779us, non-child: 29.779us, % non-child: 100.00%) - BytesSent: 256.00 B (256) - NetworkThroughput(*): 242.85 KB/sec - OverallThroughput: 8.20 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 40.447us - ThriftTransmitTime(*): 1.29ms - UncompressedRowBatchSize: 376.00 B (376) AGGREGATION_NODE (id=1):(Total: 8s542ms, non-child: 7s138ms, % non-child: 83.56%) ExecOption: Codegen Enabled - BuildTime: 6s696ms - GetNewBlockTime: 83.1us - GetResultsTime: 23.257us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 1.977us HDFS_SCAN_NODE (id=0):(Total: 1s404ms, non-child: 1s404ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB Hdfs Read Thread Concurrency Bucket: 0:52.94% 1:35.29% 2:11.76% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:53 ExecOption: Codegen enabled: 53 out of 53 BytesRead(500.0ms): 0, 710.92 MB, 950.92 MB, 1.27 GB, 1.61 GB, 1.97 GB, 2.30 GB, 2.65 GB, 2.91 GB, 3.27 GB, 3.68 GB, 4.02 GB, 4.36 GB, 4.70 GB, 5.03 GB, 5.31 GB, 5.58 GB, 5.67 GB - AverageHdfsReadThreadConcurrency: 0.59 - AverageScannerThreadConcurrency: 21.82 - BytesRead: 5.67 GB (6085373526) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.67 GB (6085367382) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.67 GB (6085367382) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 702.06 MB (736161792) - PerReadThreadRawHdfsThroughput: 1.49 GB/sec - RemoteScanRanges: 24 - RowsRead: 260.57M (260565882) - RowsReturned: 260.57M (260565882) - RowsReturnedRate: 185.56 M/sec - ScanRangesComplete: 53 - ScannerThreadsInvoluntaryContextSwitches: 9.54K (9537) - ScannerThreadsTotalWallClockTime: 3m4s - DelimiterParseTime: 15s029ms - MaterializeTupleTime(*): 11s170ms - ScannerThreadsSysTime: 1s401ms - ScannerThreadsUserTime: 30s346ms - ScannerThreadsVoluntaryContextSwitches: 283.13K (283129) - TotalRawHdfsReadTime(*): 3s797ms - TotalReadThroughput: 644.83 MB/sec Instance 634de4e3144b73be:966fad5b7fccc97 (host=a1733.halxg.cloudera.com:22000):(Total: 7s185ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB MemoryUsage(500.0ms): 46.63 MB, 554.36 MB, 568.33 MB, 560.36 MB, 560.39 MB, 552.36 MB, 560.33 MB, 520.36 MB, 336.36 MB, 510.33 MB, 408.20 MB, 368.14 MB, 368.11 MB, 284.11 MB ThreadUsage(500.0ms): 24, 24, 24, 24, 24, 24, 24, 24, 24, 23, 19, 16, 16, 15 - AverageThreadTokens: 21.79 - PeakMemoryUsage: 738.39 MB (774254592) - PerHostPeakMemUsage: 744.66 MB (780836864) - PrepareTime: 204.385ms - RowsProduced: 18 - TotalCpuTime: 2m26s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.72ms - TotalStorageWaitTime: 1s472ms CodeGen:(Total: 470.885ms, non-child: 470.885ms, % non-child: 100.00%) - CodegenTime: 2.667ms - CompileTime: 41.311ms - LoadTime: 72.552us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 228.577ms - PrepareTime: 199.559ms DataStreamSender (dst_id=3):(Total: 61.28us, non-child: 61.28us, % non-child: 100.00%) - BytesSent: 256.00 B (256) - NetworkThroughput(*): 308.15 KB/sec - OverallThroughput: 4.00 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 34.767us - ThriftTransmitTime(*): 811.282us - UncompressedRowBatchSize: 376.00 B (376) AGGREGATION_NODE (id=1):(Total: 7s184ms, non-child: 6s078ms, % non-child: 84.60%) ExecOption: Codegen Enabled - BuildTime: 5s687ms - GetNewBlockTime: 97.277us - GetResultsTime: 24.292us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 18 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.21us HDFS_SCAN_NODE (id=0):(Total: 1s106ms, non-child: 1s106ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB Hdfs Read Thread Concurrency Bucket: 0:57.14% 1:28.57% 2:7.143% 3:0% 4:0% 5:7.143% 6:0% File Formats: TEXT/NONE:94 ExecOption: Codegen enabled: 94 out of 94 BytesRead(500.0ms): 738.00 B, 761.68 MB, 1.11 GB, 1.47 GB, 1.84 GB, 2.17 GB, 2.55 GB, 2.85 GB, 3.03 GB, 3.52 GB, 3.78 GB, 4.13 GB, 4.47 GB, 4.74 GB - AverageHdfsReadThreadConcurrency: 0.79 - AverageScannerThreadConcurrency: 20.79 - BytesRead: 4.82 GB (5178211951) - BytesReadDataNodeCache: 0 - BytesReadLocal: 4.82 GB (5178198639) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 4.82 GB (5178198639) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 732.07 MB (767627264) - PerReadThreadRawHdfsThroughput: 1.51 GB/sec - RemoteScanRanges: 52 - RowsRead: 221.72M (221718312) - RowsReturned: 221.72M (221718312) - RowsReturnedRate: 200.44 M/sec - ScanRangesComplete: 94 - ScannerThreadsInvoluntaryContextSwitches: 7.45K (7450) - ScannerThreadsTotalWallClockTime: 2m20s - DelimiterParseTime: 11s822ms - MaterializeTupleTime(*): 8s782ms - ScannerThreadsSysTime: 1s118ms - ScannerThreadsUserTime: 24s153ms - ScannerThreadsVoluntaryContextSwitches: 242.25K (242248) - TotalRawHdfsReadTime(*): 3s200ms - TotalReadThroughput: 692.90 MB/sec [localhost:21000] > select count(distinct y) from billion_rows_text; +-------------------+ | count(distinct y) | +-------------------+ | 19 | +-------------------+ [localhost:21000] > profile; Query Runtime Profile: Query (id=58480b384a1190b6:52ee7a461f23d1bc): Summary: Session ID: 8f4cad0b2b385c70:b41743dae24d22ad Session Type: BEESWAX Start Time: 2015-01-12 15:15:35.355504000 End Time: 2015-01-12 15:15:45.192475000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) User: jrussell Connected User: jrussell Delegated User: Network Address: 127.0.0.1:38627 Default Db: oreilly Sql Statement: select count(distinct y) from billion_rows_text Coordinator: a1730.halxg.cloudera.com:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=3.88GB VCores=2 WARNING: The following tables are missing relevant table and/or column statistics. oreilly.billion_rows_text F02:PLAN FRAGMENT [UNPARTITIONED] 06:AGGREGATE [FINALIZE] | output: count:merge(y) | hosts=4 per-host-mem=unavailable | tuple-ids=2 row-size=8B cardinality=1 | 05:EXCHANGE [UNPARTITIONED] hosts=4 per-host-mem=unavailable tuple-ids=2 row-size=8B cardinality=1 F01:PLAN FRAGMENT [HASH(y)] DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=05, UNPARTITIONED] 02:AGGREGATE | output: count(y) | hosts=4 per-host-mem=10.00MB | tuple-ids=2 row-size=8B cardinality=1 | 04:AGGREGATE | group by: y | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=8B cardinality=unavailable | 03:EXCHANGE [HASH(y)] hosts=4 per-host-mem=0B tuple-ids=1 row-size=8B cardinality=unavailable F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=03, HASH(y)] 01:AGGREGATE | group by: y | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=8B cardinality=unavailable | 00:SCAN HDFS [oreilly.billion_rows_text, RANDOM] partitions=1/1 files=113 size=21.75GB table stats: unavailable column stats: unavailable hosts=4 per-host-mem=3.75GB tuple-ids=0 row-size=8B cardinality=unavailable ---------------- Estimated Per-Host Mem: 4160749568 Estimated Per-Host VCores: 2 Tables Missing Stats: oreilly.billion_rows_text Admission result: Admitted immediately Request Pool: root.jrussell ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail -------------------------------------------------------------------------------------------------------------------- 06:AGGREGATE 1 199.386ms 199.386ms 1 1 28.00 KB -1.00 B FINALIZE 05:EXCHANGE 1 36.432us 36.432us 4 1 0 -1.00 B UNPARTITIONED 02:AGGREGATE 4 194.731ms 197.911ms 4 1 12.00 KB 10.00 MB 04:AGGREGATE 4 2.682ms 2.755ms 19 -1 6.27 MB 128.00 MB 03:EXCHANGE 4 115.116us 164.98us 76 -1 0 0 HASH(y) 01:AGGREGATE 4 7s036ms 7s575ms 76 -1 6.31 MB 128.00 MB 00:SCAN HDFS 4 1s310ms 1s427ms 1.00B -1 730.06 MB 3.75 GB oreilly.billion_rows_text Query Timeline: 9s841ms - Start execution: 64.302us (64.302us) - Planning finished: 27.916ms (27.851ms) - Submit for admission: 37.682ms (9.766ms) - Completed admission: 37.793ms (110.662us) - Ready to start remote fragments: 237.907ms (200.114ms) - Remote fragments started: 681.784ms (443.876ms) - Rows available: 9s739ms (9s058ms) - First row fetched: 9s834ms (94.449ms) - Unregister query: 9s836ms (2.538ms) ImpalaServer: - ClientFetchWaitTimer: 95.779ms - RowMaterializationTimer: 7.780us Execution Profile 58480b384a1190b6:52ee7a461f23d1bc:(Total: 9s702ms, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 8, last: 0.240799, min: 0.196049, max: 0.240799, mean: 0.205444, stddev: 0.013657 Per Node Peak Memory Usage: a1733.halxg.cloudera.com:22000(690.66 MB) a1731.halxg.cloudera.com:22000(742.63 MB) a1732.halxg.cloudera.com:22000(722.63 MB) a1730.halxg.cloudera.com:22000(470.43 MB) - FinalizationTimer: 0ns Coordinator Fragment F02:(Total: 9s055ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB ThreadUsage(500.0ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 40.00 KB (40960) - PerHostPeakMemUsage: 0 - PrepareTime: 199.314ms - RowsProduced: 1 - TotalCpuTime: 940.736ms - TotalNetworkReceiveTime: 8s855ms - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 401.58ms, non-child: 401.58ms, % non-child: 100.00%) - CodegenTime: 526.458us - CompileTime: 7.241ms - LoadTime: 76.417us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 195.0ms - PrepareTime: 197.310ms AGGREGATION_NODE (id=6):(Total: 9s054ms, non-child: 199.386ms, % non-child: 2.20%) ExecOption: Codegen Enabled - BuildTime: 2.425us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 28.00 KB (28672) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 EXCHANGE_NODE (id=5):(Total: 8s855ms, non-child: 36.432us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 64.00 B (64) - ConvertRowBatchTime: 5.681us - DeserializeRowBatchTimer: 14.230us - FirstBatchArrivalWaitTime: 8s853ms - PeakMemoryUsage: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F01:(Total: 9s248ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:9s301ms max:9s306ms mean: 9s303ms stddev:1.784ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 4 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6607872) - PerHostPeakMemUsage: 656.59 MB (688483328) - PrepareTime: 197.0ms - RowsProduced: 1 - TotalCpuTime: 450.31ms - TotalNetworkReceiveTime: 9s050ms - TotalNetworkSendTime: 325.656us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 445.447ms, non-child: 445.447ms, % non-child: 100.00%) - CodegenTime: 2.246ms - CompileTime: 31.159ms - LoadTime: 79.231us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 219.913ms - PrepareTime: 192.959ms DataStreamSender (dst_id=5):(Total: 33.929us, non-child: 33.929us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 68.60 KB/sec - OverallThroughput: 464.73 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 5.38us - ThriftTransmitTime(*): 251.18us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s247ms, non-child: 194.731ms, % non-child: 2.11%) - BuildTime: 5.354us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s053ms, non-child: 2.682ms, % non-child: 0.03%) - BuildTime: 15.232us - GetNewBlockTime: 87.252us - GetResultsTime: 6.950us - HashBuckets: 4.35K (4352) - LargestPartitionPercent: 8 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6574080) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.46us EXCHANGE_NODE (id=3):(Total: 9s050ms, non-child: 9s050ms, % non-child: 100.00%) - BytesReceived: 274.00 B (274) - ConvertRowBatchTime: 9.313us - DeserializeRowBatchTimer: 31.933us - FirstBatchArrivalWaitTime: 7s388ms - PeakMemoryUsage: 0 - RowsReturned: 19 - RowsReturnedRate: 1.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F00:(Total: 8s348ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 4.82 GB, max: 5.67 GB, avg: 5.44 GB, stddev: 364.68 MB completion times: min:7s436ms max:9s058ms mean: 8s414ms stddev:600.535ms execution rates: min:632.93 MB/sec max:682.41 MB/sec mean:662.22 MB/sec stddev:18.18 MB/sec num instances: 4 - AverageThreadTokens: 19.63 - PeakMemoryUsage: 650.31 MB (681897984) - PerHostPeakMemUsage: 656.59 MB (688483328) - PrepareTime: 210.392ms - RowsProduced: 19 - TotalCpuTime: 2m45s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.280ms - TotalStorageWaitTime: 1s830ms CodeGen:(Total: 481.582ms, non-child: 481.582ms, % non-child: 100.00%) - CodegenTime: 2.621ms - CompileTime: 42.505ms - LoadTime: 76.463us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 231.983ms - PrepareTime: 205.684ms DataStreamSender (dst_id=3):(Total: 37.48us, non-child: 37.48us, % non-child: 100.00%) - BytesSent: 274.00 B (274) - NetworkThroughput(*): 279.29 KB/sec - OverallThroughput: 7.07 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 38.696us - ThriftTransmitTime(*): 959.920us - UncompressedRowBatchSize: 396.00 B (396) AGGREGATION_NODE (id=1):(Total: 8s347ms, non-child: 7s036ms, % non-child: 84.30%) - BuildTime: 6s604ms - GetNewBlockTime: 110.465us - GetResultsTime: 25.976us - HashBuckets: 15.36K (15360) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.100us HDFS_SCAN_NODE (id=0):(Total: 1s310ms, non-child: 1s310ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.58 - AverageScannerThreadConcurrency: 19.80 - BytesRead: 5.44 GB (5838667196) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.44 GB (5838661564) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.44 GB (5838661564) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 21 - PeakMemoryUsage: 644.01 MB (675295232) - PerReadThreadRawHdfsThroughput: 1.59 GB/sec - RemoteScanRanges: 22 - RowsRead: 250.00M (250000000) - RowsReturned: 250.00M (250000000) - RowsReturnedRate: 191.33 M/sec - ScanRangesComplete: 67 - ScannerThreadsInvoluntaryContextSwitches: 9.14K (9144) - ScannerThreadsTotalWallClockTime: 2m38s - DelimiterParseTime: 13s598ms - MaterializeTupleTime(*): 9s155ms - ScannerThreadsSysTime: 1s371ms - ScannerThreadsUserTime: 26s636ms - ScannerThreadsVoluntaryContextSwitches: 270.19K (270186) - TotalRawHdfsReadTime(*): 3s419ms - TotalReadThroughput: 660.71 MB/sec Fragment F01: Instance 58480b384a1190b6:52ee7a461f23d1c0 (host=a1732.halxg.cloudera.com:22000):(Total: 9s256ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 722.63 MB (757735424) - PrepareTime: 200.191ms - RowsProduced: 1 - TotalCpuTime: 445.484ms - TotalNetworkReceiveTime: 9s055ms - TotalNetworkSendTime: 356.971us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 440.892ms, non-child: 440.892ms, % non-child: 100.00%) - CodegenTime: 2.243ms - CompileTime: 30.610ms - LoadTime: 81.199us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 212.739ms - PrepareTime: 196.77ms DataStreamSender (dst_id=5):(Total: 33.143us, non-child: 33.143us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 50.82 KB/sec - OverallThroughput: 471.44 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 6.159us - ThriftTransmitTime(*): 307.466us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s256ms, non-child: 197.911ms, % non-child: 2.14%) ExecOption: Codegen Enabled - BuildTime: 8.729us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s058ms, non-child: 2.724ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 11.249us - GetNewBlockTime: 95.11us - GetResultsTime: 9.48us - HashBuckets: 6.14K (6144) - LargestPartitionPercent: 4 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 6 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 1.990us EXCHANGE_NODE (id=3):(Total: 9s055ms, non-child: 55.692us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86.00 B, 86.00 B, 172.00 B, 258.00 B - BytesReceived: 344.00 B (344) - ConvertRowBatchTime: 9.216us - DeserializeRowBatchTimer: 40.968us - FirstBatchArrivalWaitTime: 7s393ms - PeakMemoryUsage: 0 - RowsReturned: 24 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 58480b384a1190b6:52ee7a461f23d1bf (host=a1733.halxg.cloudera.com:22000):(Total: 9s254ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6606848) - PerHostPeakMemUsage: 690.66 MB (724213760) - PrepareTime: 197.486ms - RowsProduced: 1 - TotalCpuTime: 444.549ms - TotalNetworkReceiveTime: 9s055ms - TotalNetworkSendTime: 444.537us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 439.905ms, non-child: 439.905ms, % non-child: 100.00%) - CodegenTime: 2.200ms - CompileTime: 30.993ms - LoadTime: 88.532us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 213.995ms - PrepareTime: 193.524ms DataStreamSender (dst_id=5):(Total: 39.153us, non-child: 39.153us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 46.68 KB/sec - OverallThroughput: 399.08 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 6.768us - ThriftTransmitTime(*): 334.739us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s253ms, non-child: 195.270ms, % non-child: 2.11%) ExecOption: Codegen Enabled - BuildTime: 2.303us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s058ms, non-child: 2.645ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 14.257us - GetNewBlockTime: 85.433us - GetResultsTime: 3.571us - HashBuckets: 2.05K (2048) - LargestPartitionPercent: 12 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6569984) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 2 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.28us EXCHANGE_NODE (id=3):(Total: 9s055ms, non-child: 113.658us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 34.00 B, 34.00 B, 68.00 B - BytesReceived: 136.00 B (136) - ConvertRowBatchTime: 8.37us - DeserializeRowBatchTimer: 27.753us - FirstBatchArrivalWaitTime: 7s394ms - PeakMemoryUsage: 0 - RowsReturned: 8 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 58480b384a1190b6:52ee7a461f23d1be (host=a1731.halxg.cloudera.com:22000):(Total: 9s252ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.29 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6602752) - PerHostPeakMemUsage: 742.63 MB (778706944) - PrepareTime: 195.595ms - RowsProduced: 1 - TotalCpuTime: 443.558ms - TotalNetworkReceiveTime: 9s055ms - TotalNetworkSendTime: 309.205us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 439.102ms, non-child: 439.102ms, % non-child: 100.00%) - CodegenTime: 2.212ms - CompileTime: 31.52ms - LoadTime: 73.93us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 215.22ms - PrepareTime: 191.584ms DataStreamSender (dst_id=5):(Total: 29.984us, non-child: 29.984us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 74.82 KB/sec - OverallThroughput: 521.11 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.815us - ThriftTransmitTime(*): 208.828us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s251ms, non-child: 193.376ms, % non-child: 2.09%) ExecOption: Codegen Enabled - BuildTime: 5.834us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s058ms, non-child: 2.604ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 23.475us - GetNewBlockTime: 80.793us - GetResultsTime: 5.519us - HashBuckets: 4.10K (4096) - LargestPartitionPercent: 10 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6569984) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.65us EXCHANGE_NODE (id=3):(Total: 9s055ms, non-child: 127.19us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 73.00 B, 73.00 B, 219.00 B - BytesReceived: 292.00 B (292) - ConvertRowBatchTime: 9.426us - DeserializeRowBatchTimer: 28.328us - FirstBatchArrivalWaitTime: 7s394ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 58480b384a1190b6:52ee7a461f23d1c1 (host=a1730.halxg.cloudera.com:22000):(Total: 9s230ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 470.43 MB (493277184) - PrepareTime: 194.729ms - RowsProduced: 1 - TotalCpuTime: 466.531ms - TotalNetworkReceiveTime: 9s034ms - TotalNetworkSendTime: 191.911us - TotalStorageWaitTime: 0ns CodeGen:(Total: 461.887ms, non-child: 461.887ms, % non-child: 100.00%) - CodegenTime: 2.327ms - CompileTime: 31.980ms - LoadTime: 74.100us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 237.897ms - PrepareTime: 190.651ms DataStreamSender (dst_id=5):(Total: 33.437us, non-child: 33.437us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 102.10 KB/sec - OverallThroughput: 467.30 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.413us - ThriftTransmitTime(*): 153.42us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s229ms, non-child: 192.368ms, % non-child: 2.08%) ExecOption: Codegen Enabled - BuildTime: 4.553us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 9s037ms, non-child: 2.755ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 11.948us - GetNewBlockTime: 87.772us - GetResultsTime: 9.665us - HashBuckets: 5.12K (5120) - LargestPartitionPercent: 8 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 6 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.101us EXCHANGE_NODE (id=3):(Total: 9s034ms, non-child: 164.98us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 81.00 B, 81.00 B, 243.00 B - BytesReceived: 324.00 B (324) - ConvertRowBatchTime: 10.574us - DeserializeRowBatchTimer: 30.684us - FirstBatchArrivalWaitTime: 7s372ms - PeakMemoryUsage: 0 - RowsReturned: 24 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Fragment F00: Instance 58480b384a1190b6:52ee7a461f23d1c5 (host=a1730.halxg.cloudera.com:22000):(Total: 9s004ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB MemoryUsage(500.0ms): 0, 178.14 MB, 368.11 MB, 362.09 MB, 360.11 MB, 344.20 MB, 312.14 MB, 336.09 MB, 368.14 MB, 376.14 MB, 356.11 MB, 376.11 MB, 328.14 MB, 360.14 MB, 360.11 MB, 376.09 MB, 344.11 MB, 312.06 MB, 111.92 MB ThreadUsage(500.0ms): 1, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 14, 8 - AverageThreadTokens: 14.68 - PeakMemoryUsage: 464.14 MB (486682624) - PerHostPeakMemUsage: 470.43 MB (493277184) - PrepareTime: 239.24ms - RowsProduced: 19 - TotalCpuTime: 2m14s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.338ms - TotalStorageWaitTime: 939.373ms CodeGen:(Total: 525.555ms, non-child: 525.555ms, % non-child: 100.00%) - CodegenTime: 2.664ms - CompileTime: 45.576ms - LoadTime: 82.935us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 244.343ms - PrepareTime: 234.210ms DataStreamSender (dst_id=3):(Total: 33.886us, non-child: 33.886us, % non-child: 100.00%) - BytesSent: 274.00 B (274) - NetworkThroughput(*): 272.56 KB/sec - OverallThroughput: 7.71 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 40.947us - ThriftTransmitTime(*): 981.713us - UncompressedRowBatchSize: 396.00 B (396) AGGREGATION_NODE (id=1):(Total: 9s003ms, non-child: 7s575ms, % non-child: 84.14%) ExecOption: Codegen Enabled - BuildTime: 7s093ms - GetNewBlockTime: 149.187us - GetResultsTime: 23.994us - HashBuckets: 15.36K (15360) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.263us HDFS_SCAN_NODE (id=0):(Total: 1s427ms, non-child: 1s427ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB Hdfs Read Thread Concurrency Bucket: 0:66.67% 1:11.11% 2:16.67% 3:5.556% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:59 ExecOption: Codegen enabled: 59 out of 59 BytesRead(500.0ms): 131.74 MB, 621.24 MB, 958.76 MB, 1.25 GB, 1.55 GB, 1.86 GB, 2.21 GB, 2.57 GB, 2.91 GB, 3.23 GB, 3.57 GB, 3.86 GB, 4.22 GB, 4.54 GB, 4.85 GB, 5.14 GB, 5.43 GB, 5.57 GB - AverageHdfsReadThreadConcurrency: 0.61 - AverageScannerThreadConcurrency: 14.44 - BytesRead: 5.60 GB (6012072860) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.60 GB (6012070812) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.60 GB (6012070812) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 15 - PeakMemoryUsage: 457.84 MB (480079872) - PerReadThreadRawHdfsThroughput: 1.62 GB/sec - RemoteScanRanges: 8 - RowsRead: 257.43M (257426663) - RowsReturned: 257.43M (257426663) - RowsReturnedRate: 180.28 M/sec - ScanRangesComplete: 59 - ScannerThreadsInvoluntaryContextSwitches: 11.30K (11299) - ScannerThreadsTotalWallClockTime: 2m6s - DelimiterParseTime: 14s265ms - MaterializeTupleTime(*): 9s717ms - ScannerThreadsSysTime: 1s373ms - ScannerThreadsUserTime: 28s338ms - ScannerThreadsVoluntaryContextSwitches: 283.39K (283394) - TotalRawHdfsReadTime(*): 3s450ms - TotalReadThroughput: 633.51 MB/sec Instance 58480b384a1190b6:52ee7a461f23d1c2 (host=a1731.halxg.cloudera.com:22000):(Total: 8s600ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB MemoryUsage(500.0ms): 0, 436.36 MB, 544.36 MB, 540.36 MB, 560.36 MB, 560.36 MB, 552.33 MB, 568.36 MB, 520.36 MB, 296.36 MB, 592.33 MB, 552.33 MB, 568.36 MB, 548.33 MB, 560.36 MB, 512.30 MB, 456.24 MB, 232.22 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 23, 21, 18 - AverageThreadTokens: 22.17 - PeakMemoryUsage: 736.36 MB (772124672) - PerHostPeakMemUsage: 742.63 MB (778706944) - PrepareTime: 202.553ms - RowsProduced: 19 - TotalCpuTime: 3m13s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.355ms - TotalStorageWaitTime: 1s303ms CodeGen:(Total: 467.377ms, non-child: 467.377ms, % non-child: 100.00%) - CodegenTime: 2.539ms - CompileTime: 41.520ms - LoadTime: 71.41us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 226.442ms - PrepareTime: 197.981ms DataStreamSender (dst_id=3):(Total: 38.631us, non-child: 38.631us, % non-child: 100.00%) - BytesSent: 274.00 B (274) - NetworkThroughput(*): 262.68 KB/sec - OverallThroughput: 6.76 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 39.492us - ThriftTransmitTime(*): 1.18ms - UncompressedRowBatchSize: 396.00 B (396) AGGREGATION_NODE (id=1):(Total: 8s599ms, non-child: 7s299ms, % non-child: 84.89%) ExecOption: Codegen Enabled - BuildTime: 6s872ms - GetNewBlockTime: 89.784us - GetResultsTime: 25.505us - HashBuckets: 15.36K (15360) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.108us HDFS_SCAN_NODE (id=0):(Total: 1s299ms, non-child: 1s299ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB Hdfs Read Thread Concurrency Bucket: 0:70.59% 1:17.65% 2:5.882% 3:0% 4:5.882% 5:0% 6:0% File Formats: TEXT/NONE:53 ExecOption: Codegen enabled: 53 out of 53 BytesRead(500.0ms): 395.13 MB, 819.13 MB, 1.13 GB, 1.48 GB, 1.81 GB, 2.15 GB, 2.51 GB, 2.81 GB, 2.95 GB, 3.59 GB, 3.88 GB, 4.22 GB, 4.58 GB, 4.92 GB, 5.22 GB, 5.50 GB, 5.62 GB - AverageHdfsReadThreadConcurrency: 0.53 - AverageScannerThreadConcurrency: 22.41 - BytesRead: 5.67 GB (6085373526) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.67 GB (6085367382) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.67 GB (6085367382) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 730.06 MB (765521920) - PerReadThreadRawHdfsThroughput: 1.62 GB/sec - RemoteScanRanges: 24 - RowsRead: 260.57M (260565882) - RowsReturned: 260.57M (260565882) - RowsReturnedRate: 200.51 M/sec - ScanRangesComplete: 53 - ScannerThreadsInvoluntaryContextSwitches: 8.94K (8941) - ScannerThreadsTotalWallClockTime: 3m5s - DelimiterParseTime: 14s235ms - MaterializeTupleTime(*): 9s631ms - ScannerThreadsSysTime: 1s222ms - ScannerThreadsUserTime: 27s717ms - ScannerThreadsVoluntaryContextSwitches: 277.33K (277328) - TotalRawHdfsReadTime(*): 3s507ms - TotalReadThroughput: 677.55 MB/sec Instance 58480b384a1190b6:52ee7a461f23d1c4 (host=a1732.halxg.cloudera.com:22000):(Total: 8s424ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB MemoryUsage(500.0ms): 6.29 MB, 620.33 MB, 568.33 MB, 560.36 MB, 522.36 MB, 576.33 MB, 560.36 MB, 532.33 MB, 424.33 MB, 480.37 MB, 520.36 MB, 560.36 MB, 528.32 MB, 520.30 MB, 496.30 MB, 488.25 MB, 280.19 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 22, 22, 21, 18 - AverageThreadTokens: 21.88 - PeakMemoryUsage: 716.36 MB (751153152) - PerHostPeakMemUsage: 722.63 MB (757735424) - PrepareTime: 201.7ms - RowsProduced: 19 - TotalCpuTime: 3m6s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.210ms - TotalStorageWaitTime: 1s382ms CodeGen:(Total: 467.778ms, non-child: 467.778ms, % non-child: 100.00%) - CodegenTime: 2.665ms - CompileTime: 41.474ms - LoadTime: 74.23us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 228.604ms - PrepareTime: 196.267ms DataStreamSender (dst_id=3):(Total: 37.194us, non-child: 37.194us, % non-child: 100.00%) - BytesSent: 274.00 B (274) - NetworkThroughput(*): 292.56 KB/sec - OverallThroughput: 7.03 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 34.815us - ThriftTransmitTime(*): 914.620us - UncompressedRowBatchSize: 396.00 B (396) AGGREGATION_NODE (id=1):(Total: 8s422ms, non-child: 7s025ms, % non-child: 83.41%) ExecOption: Codegen Enabled - BuildTime: 6s597ms - GetNewBlockTime: 105.758us - GetResultsTime: 24.768us - HashBuckets: 15.36K (15360) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.56us HDFS_SCAN_NODE (id=0):(Total: 1s397ms, non-child: 1s397ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB Hdfs Read Thread Concurrency Bucket: 0:43.75% 1:37.5% 2:12.5% 3:6.25% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:64 ExecOption: Codegen enabled: 64 out of 64 BytesRead(500.0ms): 0, 709.58 MB, 1013.58 MB, 1.33 GB, 1.63 GB, 2.02 GB, 2.35 GB, 2.69 GB, 2.94 GB, 3.29 GB, 3.70 GB, 4.09 GB, 4.41 GB, 4.76 GB, 5.08 GB, 5.43 GB, 5.61 GB - AverageHdfsReadThreadConcurrency: 0.81 - AverageScannerThreadConcurrency: 22.19 - BytesRead: 5.66 GB (6079010449) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.66 GB (6079009425) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.66 GB (6079009425) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 710.06 MB (744550400) - PerReadThreadRawHdfsThroughput: 1.57 GB/sec - RemoteScanRanges: 4 - RowsRead: 260.29M (260289143) - RowsReturned: 260.29M (260289143) - RowsReturnedRate: 186.31 M/sec - ScanRangesComplete: 64 - ScannerThreadsInvoluntaryContextSwitches: 8.86K (8856) - ScannerThreadsTotalWallClockTime: 2m58s - DelimiterParseTime: 14s015ms - MaterializeTupleTime(*): 9s333ms - ScannerThreadsSysTime: 1s849ms - ScannerThreadsUserTime: 27s241ms - ScannerThreadsVoluntaryContextSwitches: 281.65K (281655) - TotalRawHdfsReadTime(*): 3s610ms - TotalReadThroughput: 675.46 MB/sec Instance 58480b384a1190b6:52ee7a461f23d1c3 (host=a1733.halxg.cloudera.com:22000):(Total: 7s363ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB MemoryUsage(500.0ms): 6.29 MB, 643.34 MB, 576.36 MB, 552.36 MB, 560.36 MB, 556.33 MB, 536.48 MB, 544.33 MB, 456.33 MB, 402.36 MB, 440.22 MB, 392.17 MB, 360.14 MB, 312.06 MB, 208.00 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 20, 18, 16, 14, 12 - AverageThreadTokens: 19.80 - PeakMemoryUsage: 684.39 MB (717631488) - PerHostPeakMemUsage: 690.66 MB (724213760) - PrepareTime: 198.984ms - RowsProduced: 19 - TotalCpuTime: 2m27s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.219ms - TotalStorageWaitTime: 3s698ms CodeGen:(Total: 465.617ms, non-child: 465.617ms, % non-child: 100.00%) - CodegenTime: 2.616ms - CompileTime: 41.448ms - LoadTime: 77.853us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 228.543ms - PrepareTime: 194.278ms DataStreamSender (dst_id=3):(Total: 38.484us, non-child: 38.484us, % non-child: 100.00%) - BytesSent: 274.00 B (274) - NetworkThroughput(*): 289.37 KB/sec - OverallThroughput: 6.79 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 39.531us - ThriftTransmitTime(*): 924.692us - UncompressedRowBatchSize: 396.00 B (396) AGGREGATION_NODE (id=1):(Total: 7s362ms, non-child: 6s243ms, % non-child: 84.81%) ExecOption: Codegen Enabled - BuildTime: 5s855ms - GetNewBlockTime: 97.131us - GetResultsTime: 29.638us - HashBuckets: 15.36K (15360) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 19 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 1.975us HDFS_SCAN_NODE (id=0):(Total: 1s118ms, non-child: 1s118ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB Hdfs Read Thread Concurrency Bucket: 0:64.29% 1:35.71% 2:0% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:94 ExecOption: Codegen enabled: 94 out of 94 BytesRead(500.0ms): 0, 765.19 MB, 1.02 GB, 1.34 GB, 1.70 GB, 2.03 GB, 2.36 GB, 2.72 GB, 3.00 GB, 3.28 GB, 3.68 GB, 3.94 GB, 4.27 GB, 4.57 GB, 4.81 GB - AverageHdfsReadThreadConcurrency: 0.36 - AverageScannerThreadConcurrency: 20.14 - BytesRead: 4.82 GB (5178211951) - BytesReadDataNodeCache: 0 - BytesReadLocal: 4.82 GB (5178198639) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 4.82 GB (5178198639) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 678.09 MB (711028736) - PerReadThreadRawHdfsThroughput: 1.55 GB/sec - RemoteScanRanges: 52 - RowsRead: 221.72M (221718312) - RowsReturned: 221.72M (221718312) - RowsReturnedRate: 198.21 M/sec - ScanRangesComplete: 94 - ScannerThreadsInvoluntaryContextSwitches: 7.48K (7483) - ScannerThreadsTotalWallClockTime: 2m23s - DelimiterParseTime: 11s876ms - MaterializeTupleTime(*): 7s938ms - ScannerThreadsSysTime: 1s040ms - ScannerThreadsUserTime: 23s247ms - ScannerThreadsVoluntaryContextSwitches: 238.37K (238369) - TotalRawHdfsReadTime(*): 3s110ms - TotalReadThroughput: 656.31 MB/sec [localhost:21000] > select count(distinct z) from billion_rows_text; +-------------------+ | count(distinct z) | +-------------------+ | 20 | +-------------------+ [localhost:21000] > profile; Query Runtime Profile: Query (id=20446ca5e8a25a1c:4e662b8544c2548a): Summary: Session ID: 8f4cad0b2b385c70:b41743dae24d22ad Session Type: BEESWAX Start Time: 2015-01-12 15:15:54.404489000 End Time: 2015-01-12 15:16:03.978485000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) User: jrussell Connected User: jrussell Delegated User: Network Address: 127.0.0.1:38627 Default Db: oreilly Sql Statement: select count(distinct z) from billion_rows_text Coordinator: a1730.halxg.cloudera.com:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=3.88GB VCores=2 WARNING: The following tables are missing relevant table and/or column statistics. oreilly.billion_rows_text F02:PLAN FRAGMENT [UNPARTITIONED] 06:AGGREGATE [FINALIZE] | output: count:merge(z) | hosts=4 per-host-mem=unavailable | tuple-ids=2 row-size=8B cardinality=1 | 05:EXCHANGE [UNPARTITIONED] hosts=4 per-host-mem=unavailable tuple-ids=2 row-size=8B cardinality=1 F01:PLAN FRAGMENT [HASH(z)] DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=05, UNPARTITIONED] 02:AGGREGATE | output: count(z) | hosts=4 per-host-mem=10.00MB | tuple-ids=2 row-size=8B cardinality=1 | 04:AGGREGATE | group by: z | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=8B cardinality=unavailable | 03:EXCHANGE [HASH(z)] hosts=4 per-host-mem=0B tuple-ids=1 row-size=8B cardinality=unavailable F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=03, HASH(z)] 01:AGGREGATE | group by: z | hosts=4 per-host-mem=128.00MB | tuple-ids=1 row-size=8B cardinality=unavailable | 00:SCAN HDFS [oreilly.billion_rows_text, RANDOM] partitions=1/1 files=113 size=21.75GB table stats: unavailable column stats: unavailable hosts=4 per-host-mem=3.75GB tuple-ids=0 row-size=8B cardinality=unavailable ---------------- Estimated Per-Host Mem: 4160749568 Estimated Per-Host VCores: 2 Tables Missing Stats: oreilly.billion_rows_text Admission result: Admitted immediately Request Pool: root.jrussell ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail -------------------------------------------------------------------------------------------------------------------- 06:AGGREGATE 1 197.824ms 197.824ms 1 1 28.00 KB -1.00 B FINALIZE 05:EXCHANGE 1 128.186us 128.186us 4 1 0 -1.00 B UNPARTITIONED 02:AGGREGATE 4 193.763ms 195.250ms 4 1 12.00 KB 10.00 MB 04:AGGREGATE 4 2.705ms 2.882ms 20 -1 6.27 MB 128.00 MB 03:EXCHANGE 4 90.404us 138.458us 80 -1 0 0 HASH(z) 01:AGGREGATE 4 6s838ms 7s325ms 80 -1 6.31 MB 128.00 MB 00:SCAN HDFS 4 1s354ms 1s475ms 1.00B -1 730.06 MB 3.75 GB oreilly.billion_rows_text Query Timeline: 9s578ms - Start execution: 63.813us (63.813us) - Planning finished: 19.957ms (19.894ms) - Submit for admission: 20.522ms (564.896us) - Completed admission: 20.591ms (68.977us) - Ready to start remote fragments: 219.121ms (198.529ms) - Remote fragments started: 621.141ms (402.20ms) - Rows available: 9s507ms (8s886ms) - First row fetched: 9s571ms (63.878ms) - Unregister query: 9s574ms (2.479ms) ImpalaServer: - ClientFetchWaitTimer: 65.217ms - RowMaterializationTimer: 6.490us Execution Profile 20446ca5e8a25a1c:4e662b8544c2548a:(Total: 9s487ms, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 8, last: 0.202064, min: 0.195285, max: 0.202156, mean: 0.199548, stddev: 0.00252568 Per Node Peak Memory Usage: a1733.halxg.cloudera.com:22000(640.66 MB) a1731.halxg.cloudera.com:22000(742.63 MB) a1732.halxg.cloudera.com:22000(686.66 MB) a1730.halxg.cloudera.com:22000(494.43 MB) - FinalizationTimer: 0ns Coordinator Fragment F02:(Total: 8s887ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB ThreadUsage(500.0ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 40.00 KB (40960) - PerHostPeakMemUsage: 0 - PrepareTime: 197.751ms - RowsProduced: 1 - TotalCpuTime: 860.967ms - TotalNetworkReceiveTime: 8s689ms - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 393.844ms, non-child: 393.844ms, % non-child: 100.00%) - CodegenTime: 515.365us - CompileTime: 6.642ms - LoadTime: 30.112us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 189.941ms - PrepareTime: 195.855ms AGGREGATION_NODE (id=6):(Total: 8s887ms, non-child: 197.824ms, % non-child: 2.23%) ExecOption: Codegen Enabled - BuildTime: 2.978us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 28.00 KB (28672) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 EXCHANGE_NODE (id=5):(Total: 8s689ms, non-child: 128.186us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 64.00 B (64) - ConvertRowBatchTime: 6.367us - DeserializeRowBatchTimer: 16.669us - FirstBatchArrivalWaitTime: 8s687ms - PeakMemoryUsage: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F01:(Total: 9s039ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:9s088ms max:9s093ms mean: 9s090ms stddev:1.700ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 4 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6609920) - PerHostPeakMemUsage: 641.10 MB (672240640) - PrepareTime: 196.9ms - RowsProduced: 1 - TotalCpuTime: 443.884ms - TotalNetworkReceiveTime: 8s842ms - TotalNetworkSendTime: 348.736us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 439.274ms, non-child: 439.274ms, % non-child: 100.00%) - CodegenTime: 2.231ms - CompileTime: 30.997ms - LoadTime: 80.689us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 214.879ms - PrepareTime: 191.999ms DataStreamSender (dst_id=5):(Total: 36.384us, non-child: 36.384us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 70.89 KB/sec - OverallThroughput: 439.21 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 4.817us - ThriftTransmitTime(*): 245.532us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s038ms, non-child: 193.763ms, % non-child: 2.14%) - BuildTime: 8.104us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s845ms, non-child: 2.705ms, % non-child: 0.03%) - BuildTime: 12.144us - GetNewBlockTime: 87.491us - GetResultsTime: 8.27us - HashBuckets: 4.61K (4608) - LargestPartitionPercent: 7 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6576128) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.39us EXCHANGE_NODE (id=3):(Total: 8s842ms, non-child: 8s842ms, % non-child: 100.00%) - BytesReceived: 232.00 B (232) - ConvertRowBatchTime: 8.739us - DeserializeRowBatchTimer: 27.555us - FirstBatchArrivalWaitTime: 7s371ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F00:(Total: 8s194ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 4.82 GB, max: 5.67 GB, avg: 5.44 GB, stddev: 364.68 MB completion times: min:7s416ms max:8s886ms mean: 8s269ms stddev:569.296ms execution rates: min:645.16 MB/sec max:715.32 MB/sec mean:673.95 MB/sec stddev:25.62 MB/sec num instances: 4 - AverageThreadTokens: 19.21 - PeakMemoryUsage: 634.82 MB (665655296) - PerHostPeakMemUsage: 641.10 MB (672240640) - PrepareTime: 200.46ms - RowsProduced: 20 - TotalCpuTime: 2m41s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.276ms - TotalStorageWaitTime: 1s390ms CodeGen:(Total: 469.406ms, non-child: 469.406ms, % non-child: 100.00%) - CodegenTime: 2.630ms - CompileTime: 41.636ms - LoadTime: 72.629us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 230.987ms - PrepareTime: 195.256ms DataStreamSender (dst_id=3):(Total: 35.869us, non-child: 35.869us, % non-child: 100.00%) - BytesSent: 232.00 B (232) - NetworkThroughput(*): 223.59 KB/sec - OverallThroughput: 6.18 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 38.942us - ThriftTransmitTime(*): 1.15ms - UncompressedRowBatchSize: 416.00 B (416) AGGREGATION_NODE (id=1):(Total: 8s192ms, non-child: 6s838ms, % non-child: 83.47%) - BuildTime: 6s413ms - GetNewBlockTime: 120.599us - GetResultsTime: 22.643us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.27us HDFS_SCAN_NODE (id=0):(Total: 1s354ms, non-child: 1s354ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.31 - AverageScannerThreadConcurrency: 19.35 - BytesRead: 5.44 GB (5838667196) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.44 GB (5838661564) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.44 GB (5838661564) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 21 - PeakMemoryUsage: 628.51 MB (659038208) - PerReadThreadRawHdfsThroughput: 1.52 GB/sec - RemoteScanRanges: 27 - RowsRead: 250.00M (250000000) - RowsReturned: 250.00M (250000000) - RowsReturnedRate: 185.00 M/sec - ScanRangesComplete: 67 - ScannerThreadsInvoluntaryContextSwitches: 8.87K (8872) - ScannerThreadsTotalWallClockTime: 2m34s - DelimiterParseTime: 13s799ms - MaterializeTupleTime(*): 8s641ms - ScannerThreadsSysTime: 1s322ms - ScannerThreadsUserTime: 26s310ms - ScannerThreadsVoluntaryContextSwitches: 272.60K (272601) - TotalRawHdfsReadTime(*): 3s574ms - TotalReadThroughput: 663.46 MB/sec Fragment F01: Instance 20446ca5e8a25a1c:4e662b8544c2548f (host=a1730.halxg.cloudera.com:22000):(Total: 9s039ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.30 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 494.43 MB (518451200) - PrepareTime: 196.248ms - RowsProduced: 1 - TotalCpuTime: 444.300ms - TotalNetworkReceiveTime: 8s842ms - TotalNetworkSendTime: 175.518us - TotalStorageWaitTime: 0ns CodeGen:(Total: 439.855ms, non-child: 439.855ms, % non-child: 100.00%) - CodegenTime: 2.299ms - CompileTime: 31.21ms - LoadTime: 83.377us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 215.285ms - PrepareTime: 192.158ms DataStreamSender (dst_id=5):(Total: 40.54us, non-child: 40.54us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 115.38 KB/sec - OverallThroughput: 390.10 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 4.153us - ThriftTransmitTime(*): 135.423us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s039ms, non-child: 193.920ms, % non-child: 2.15%) ExecOption: Codegen Enabled - BuildTime: 9.187us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s845ms, non-child: 2.882ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 11.191us - GetNewBlockTime: 97.44us - GetResultsTime: 8.560us - HashBuckets: 4.10K (4096) - LargestPartitionPercent: 10 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 1.988us EXCHANGE_NODE (id=3):(Total: 8s842ms, non-child: 50.202us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58.00 B, 116.00 B, 174.00 B - BytesReceived: 232.00 B (232) - ConvertRowBatchTime: 8.743us - DeserializeRowBatchTimer: 27.227us - FirstBatchArrivalWaitTime: 7s371ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 20446ca5e8a25a1c:4e662b8544c2548d (host=a1733.halxg.cloudera.com:22000):(Total: 9s039ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 0, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.30 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6606848) - PerHostPeakMemUsage: 640.66 MB (671784960) - PrepareTime: 196.313ms - RowsProduced: 1 - TotalCpuTime: 443.590ms - TotalNetworkReceiveTime: 8s842ms - TotalNetworkSendTime: 355.591us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 439.59ms, non-child: 439.59ms, % non-child: 100.00%) - CodegenTime: 2.210ms - CompileTime: 31.21ms - LoadTime: 73.475us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 214.329ms - PrepareTime: 192.387ms DataStreamSender (dst_id=5):(Total: 30.182us, non-child: 30.182us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 62.97 KB/sec - OverallThroughput: 517.69 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 5.145us - ThriftTransmitTime(*): 248.148us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s039ms, non-child: 194.67ms, % non-child: 2.15%) ExecOption: Codegen Enabled - BuildTime: 8.960us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s845ms, non-child: 2.655ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 16.355us - GetNewBlockTime: 93.919us - GetResultsTime: 9.449us - HashBuckets: 5.12K (5120) - LargestPartitionPercent: 5 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6569984) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.92us EXCHANGE_NODE (id=3):(Total: 8s842ms, non-child: 113.381us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58.00 B, 116.00 B, 116.00 B - BytesReceived: 232.00 B (232) - ConvertRowBatchTime: 7.376us - DeserializeRowBatchTimer: 23.299us - FirstBatchArrivalWaitTime: 7s371ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 20446ca5e8a25a1c:4e662b8544c2548c (host=a1731.halxg.cloudera.com:22000):(Total: 9s039ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.30 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 742.63 MB (778706944) - PrepareTime: 197.473ms - RowsProduced: 1 - TotalCpuTime: 444.466ms - TotalNetworkReceiveTime: 8s840ms - TotalNetworkSendTime: 486.691us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 439.864ms, non-child: 439.864ms, % non-child: 100.00%) - CodegenTime: 2.223ms - CompileTime: 30.921ms - LoadTime: 74.641us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 214.45ms - PrepareTime: 193.390ms DataStreamSender (dst_id=5):(Total: 43.289us, non-child: 43.289us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 48.06 KB/sec - OverallThroughput: 360.95 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 3.810us - ThriftTransmitTime(*): 325.141us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s038ms, non-child: 195.250ms, % non-child: 2.16%) ExecOption: Codegen Enabled - BuildTime: 7.825us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s843ms, non-child: 2.628ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 10.512us - GetNewBlockTime: 83.458us - GetResultsTime: 6.893us - HashBuckets: 5.12K (5120) - LargestPartitionPercent: 5 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.36us EXCHANGE_NODE (id=3):(Total: 8s840ms, non-child: 138.458us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58.00 B, 116.00 B, 174.00 B - BytesReceived: 232.00 B (232) - ConvertRowBatchTime: 8.712us - DeserializeRowBatchTimer: 27.834us - FirstBatchArrivalWaitTime: 7s370ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 20446ca5e8a25a1c:4e662b8544c2548e (host=a1732.halxg.cloudera.com:22000):(Total: 9s038ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500.0ms): 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.28 MB, 6.29 MB, 6.29 MB, 6.30 MB ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 6.30 MB (6610944) - PerHostPeakMemUsage: 686.66 MB (720019456) - PrepareTime: 194.3ms - RowsProduced: 1 - TotalCpuTime: 443.180ms - TotalNetworkReceiveTime: 8s843ms - TotalNetworkSendTime: 377.144us - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 - BlocksCreated: 192 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 30.16 GB (32385617920) - PeakMemoryUsage: 512.00 KB (524288) - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 438.316ms, non-child: 438.316ms, % non-child: 100.00%) - CodegenTime: 2.190ms - CompileTime: 31.26ms - LoadTime: 91.263us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 215.859ms - PrepareTime: 190.61ms DataStreamSender (dst_id=5):(Total: 32.11us, non-child: 32.11us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 57.15 KB/sec - OverallThroughput: 488.11 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 6.163us - ThriftTransmitTime(*): 273.416us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=2):(Total: 9s038ms, non-child: 191.815ms, % non-child: 2.12%) ExecOption: Codegen Enabled - BuildTime: 6.445us - GetResultsTime: 0ns - HashBuckets: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 12.00 KB (12288) - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 AGGREGATION_NODE (id=4):(Total: 8s846ms, non-child: 2.656ms, % non-child: 0.03%) ExecOption: Codegen Enabled - BuildTime: 10.519us - GetNewBlockTime: 75.544us - GetResultsTime: 7.209us - HashBuckets: 4.10K (4096) - LargestPartitionPercent: 10 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.27 MB (6578176) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 5 - RowsReturnedRate: 0 - SpilledPartitions: 0 - UnpinTime: 2.42us EXCHANGE_NODE (id=3):(Total: 8s843ms, non-child: 59.578us, % non-child: 0.00%) BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58.00 B, 58.00 B, 116.00 B - BytesReceived: 232.00 B (232) - ConvertRowBatchTime: 10.127us - DeserializeRowBatchTimer: 31.863us - FirstBatchArrivalWaitTime: 7s372ms - PeakMemoryUsage: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Fragment F00: Instance 20446ca5e8a25a1c:4e662b8544c25493 (host=a1730.halxg.cloudera.com:22000):(Total: 8s802ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB MemoryUsage(500.0ms): 0, 268.11 MB, 368.14 MB, 368.11 MB, 368.14 MB, 368.12 MB, 288.11 MB, 376.11 MB, 360.14 MB, 352.14 MB, 356.14 MB, 368.14 MB, 296.14 MB, 368.14 MB, 352.11 MB, 330.14 MB, 312.06 MB, 232.03 MB, 14.35 MB ThreadUsage(500.0ms): 1, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 14, 12, 2 - AverageThreadTokens: 14.16 - PeakMemoryUsage: 488.14 MB (511856640) - PerHostPeakMemUsage: 494.43 MB (518451200) - PrepareTime: 200.331ms - RowsProduced: 20 - TotalCpuTime: 2m10s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.430ms - TotalStorageWaitTime: 1s059ms CodeGen:(Total: 478.556ms, non-child: 478.556ms, % non-child: 100.00%) - CodegenTime: 2.750ms - CompileTime: 42.683ms - LoadTime: 69.290us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 238.894ms - PrepareTime: 195.491ms DataStreamSender (dst_id=3):(Total: 37.82us, non-child: 37.82us, % non-child: 100.00%) - BytesSent: 232.00 B (232) - NetworkThroughput(*): 217.37 KB/sec - OverallThroughput: 5.97 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 41.206us - ThriftTransmitTime(*): 1.42ms - UncompressedRowBatchSize: 416.00 B (416) AGGREGATION_NODE (id=1):(Total: 8s801ms, non-child: 7s325ms, % non-child: 83.23%) ExecOption: Codegen Enabled - BuildTime: 6s873ms - GetNewBlockTime: 103.791us - GetResultsTime: 24.53us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.53us HDFS_SCAN_NODE (id=0):(Total: 1s475ms, non-child: 1s475ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 4:10/896.81 MB 2:16/1.76 GB 1:12/1.14 GB 0:11/962.39 MB 3:10/909.05 MB Hdfs Read Thread Concurrency Bucket: 0:83.33% 1:11.11% 2:0% 3:0% 4:0% 5:5.556% 6:0% File Formats: TEXT/NONE:59 ExecOption: Codegen enabled: 59 out of 59 BytesRead(500.0ms): 218.96 MB, 634.96 MB, 975.16 MB, 1.30 GB, 1.64 GB, 1.87 GB, 2.29 GB, 2.62 GB, 2.94 GB, 3.24 GB, 3.59 GB, 3.84 GB, 4.26 GB, 4.59 GB, 4.91 GB, 5.22 GB, 5.47 GB, 5.60 GB - AverageHdfsReadThreadConcurrency: 0.39 - AverageScannerThreadConcurrency: 13.89 - BytesRead: 5.60 GB (6012072860) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.60 GB (6012070812) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.60 GB (6012070812) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 15 - PeakMemoryUsage: 481.84 MB (505245696) - PerReadThreadRawHdfsThroughput: 1.49 GB/sec - RemoteScanRanges: 10 - RowsRead: 257.43M (257426663) - RowsReturned: 257.43M (257426663) - RowsReturnedRate: 174.42 M/sec - ScanRangesComplete: 59 - ScannerThreadsInvoluntaryContextSwitches: 10.78K (10781) - ScannerThreadsTotalWallClockTime: 2m2s - DelimiterParseTime: 14s287ms - MaterializeTupleTime(*): 8s973ms - ScannerThreadsSysTime: 1s456ms - ScannerThreadsUserTime: 27s482ms - ScannerThreadsVoluntaryContextSwitches: 281.34K (281342) - TotalRawHdfsReadTime(*): 3s766ms - TotalReadThroughput: 637.06 MB/sec Instance 20446ca5e8a25a1c:4e662b8544c25490 (host=a1731.halxg.cloudera.com:22000):(Total: 8s598ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB MemoryUsage(500.0ms): 0, 288.36 MB, 552.33 MB, 568.33 MB, 560.33 MB, 560.36 MB, 560.33 MB, 560.33 MB, 528.36 MB, 320.36 MB, 608.33 MB, 544.36 MB, 556.36 MB, 540.36 MB, 536.34 MB, 492.28 MB, 456.25 MB, 200.14 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 22, 21, 17 - AverageThreadTokens: 22.06 - PeakMemoryUsage: 736.36 MB (772124672) - PerHostPeakMemUsage: 742.63 MB (778706944) - PrepareTime: 200.220ms - RowsProduced: 20 - TotalCpuTime: 3m11s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.183ms - TotalStorageWaitTime: 1s272ms CodeGen:(Total: 465.699ms, non-child: 465.699ms, % non-child: 100.00%) - CodegenTime: 2.572ms - CompileTime: 41.182ms - LoadTime: 62.30us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 227.399ms - PrepareTime: 195.508ms DataStreamSender (dst_id=3):(Total: 36.102us, non-child: 36.102us, % non-child: 100.00%) - BytesSent: 232.00 B (232) - NetworkThroughput(*): 223.17 KB/sec - OverallThroughput: 6.13 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 40.720us - ThriftTransmitTime(*): 1.15ms - UncompressedRowBatchSize: 416.00 B (416) AGGREGATION_NODE (id=1):(Total: 8s596ms, non-child: 7s155ms, % non-child: 83.23%) ExecOption: Codegen Enabled - BuildTime: 6s724ms - GetNewBlockTime: 100.100us - GetResultsTime: 25.501us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.25us HDFS_SCAN_NODE (id=0):(Total: 1s441ms, non-child: 1s441ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:6/628.00 MB 4:8/877.82 MB 2:10/1.06 GB 1:13/1.50 GB 3:16/1.64 GB Hdfs Read Thread Concurrency Bucket: 0:76.47% 1:17.65% 2:0% 3:5.882% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:53 ExecOption: Codegen enabled: 53 out of 53 BytesRead(500.0ms): 240.00 MB, 790.75 MB, 1.13 GB, 1.48 GB, 1.83 GB, 2.18 GB, 2.53 GB, 2.82 GB, 2.99 GB, 3.61 GB, 3.87 GB, 4.21 GB, 4.55 GB, 4.89 GB, 5.22 GB, 5.53 GB, 5.63 GB - AverageHdfsReadThreadConcurrency: 0.35 - AverageScannerThreadConcurrency: 22.29 - BytesRead: 5.67 GB (6085373526) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.67 GB (6085367382) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.67 GB (6085367382) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 730.06 MB (765521920) - PerReadThreadRawHdfsThroughput: 1.51 GB/sec - RemoteScanRanges: 30 - RowsRead: 260.57M (260565882) - RowsReturned: 260.57M (260565882) - RowsReturnedRate: 180.72 M/sec - ScanRangesComplete: 53 - ScannerThreadsInvoluntaryContextSwitches: 9.08K (9081) - ScannerThreadsTotalWallClockTime: 3m3s - DelimiterParseTime: 14s725ms - MaterializeTupleTime(*): 9s092ms - ScannerThreadsSysTime: 1s312ms - ScannerThreadsUserTime: 27s898ms - ScannerThreadsVoluntaryContextSwitches: 281.75K (281749) - TotalRawHdfsReadTime(*): 3s745ms - TotalReadThroughput: 678.44 MB/sec Instance 20446ca5e8a25a1c:4e662b8544c25492 (host=a1732.halxg.cloudera.com:22000):(Total: 8s032ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB MemoryUsage(500.0ms): 6.29 MB, 604.33 MB, 560.36 MB, 560.36 MB, 544.33 MB, 560.33 MB, 560.33 MB, 520.36 MB, 416.36 MB, 552.36 MB, 560.36 MB, 560.33 MB, 530.36 MB, 512.30 MB, 484.30 MB, 312.20 MB, 14.32 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 22, 22, 19, 2 - AverageThreadTokens: 20.82 - PeakMemoryUsage: 680.39 MB (713437184) - PerHostPeakMemUsage: 686.66 MB (720019456) - PrepareTime: 200.65ms - RowsProduced: 20 - TotalCpuTime: 2m56s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.230ms - TotalStorageWaitTime: 1s384ms CodeGen:(Total: 466.884ms, non-child: 466.884ms, % non-child: 100.00%) - CodegenTime: 2.608ms - CompileTime: 41.298ms - LoadTime: 93.528us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 228.795ms - PrepareTime: 195.218ms DataStreamSender (dst_id=3):(Total: 36.615us, non-child: 36.615us, % non-child: 100.00%) - BytesSent: 232.00 B (232) - NetworkThroughput(*): 212.05 KB/sec - OverallThroughput: 6.04 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 36.162us - ThriftTransmitTime(*): 1.68ms - UncompressedRowBatchSize: 416.00 B (416) AGGREGATION_NODE (id=1):(Total: 8s031ms, non-child: 6s734ms, % non-child: 83.85%) ExecOption: Codegen Enabled - BuildTime: 6s311ms - GetNewBlockTime: 169.723us - GetResultsTime: 20.587us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.25us HDFS_SCAN_NODE (id=0):(Total: 1s296ms, non-child: 1s296ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 2:15/1.43 GB 0:12/1.16 GB 1:15/1.40 GB 4:9/411.31 MB 3:13/1.26 GB Hdfs Read Thread Concurrency Bucket: 0:68.75% 1:25% 2:6.25% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:64 ExecOption: Codegen enabled: 64 out of 64 BytesRead(500.0ms): 0, 711.96 MB, 1.02 GB, 1.38 GB, 1.74 GB, 2.15 GB, 2.49 GB, 2.81 GB, 3.08 GB, 3.57 GB, 3.95 GB, 4.29 GB, 4.63 GB, 4.99 GB, 5.36 GB, 5.59 GB, 5.66 GB - AverageHdfsReadThreadConcurrency: 0.38 - AverageScannerThreadConcurrency: 21.06 - BytesRead: 5.66 GB (6079010449) - BytesReadDataNodeCache: 0 - BytesReadLocal: 5.66 GB (6079009425) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 5.66 GB (6079009425) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 674.07 MB (706809856) - PerReadThreadRawHdfsThroughput: 1.52 GB/sec - RemoteScanRanges: 5 - RowsRead: 260.29M (260289143) - RowsReturned: 260.29M (260289143) - RowsReturnedRate: 200.72 M/sec - ScanRangesComplete: 64 - ScannerThreadsInvoluntaryContextSwitches: 8.15K (8148) - ScannerThreadsTotalWallClockTime: 2m49s - DelimiterParseTime: 13s970ms - MaterializeTupleTime(*): 8s708ms - ScannerThreadsSysTime: 1s230ms - ScannerThreadsUserTime: 26s655ms - ScannerThreadsVoluntaryContextSwitches: 285.32K (285325) - TotalRawHdfsReadTime(*): 3s715ms - TotalReadThroughput: 682.05 MB/sec Instance 20446ca5e8a25a1c:4e662b8544c25491 (host=a1733.halxg.cloudera.com:22000):(Total: 7s343ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB MemoryUsage(500.0ms): 6.29 MB, 584.36 MB, 546.36 MB, 552.36 MB, 560.33 MB, 560.36 MB, 552.33 MB, 560.48 MB, 440.36 MB, 536.36 MB, 400.17 MB, 384.16 MB, 364.11 MB, 344.11 MB, 240.08 MB ThreadUsage(500.0ms): 1, 24, 24, 24, 24, 24, 24, 24, 24, 24, 18, 17, 16, 15, 14 - AverageThreadTokens: 19.80 - PeakMemoryUsage: 634.39 MB (665202688) - PerHostPeakMemUsage: 640.66 MB (671784960) - PrepareTime: 199.568ms - RowsProduced: 20 - TotalCpuTime: 2m28s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 1.260ms - TotalStorageWaitTime: 1s845ms CodeGen:(Total: 466.486ms, non-child: 466.486ms, % non-child: 100.00%) - CodegenTime: 2.590ms - CompileTime: 41.382ms - LoadTime: 65.669us - ModuleFileSize: 1.96 MB (2054956) - OptimizationTime: 228.860ms - PrepareTime: 194.807ms DataStreamSender (dst_id=3):(Total: 33.680us, non-child: 33.680us, % non-child: 100.00%) - BytesSent: 232.00 B (232) - NetworkThroughput(*): 241.78 KB/sec - OverallThroughput: 6.57 MB/sec - PeakMemoryUsage: 96.00 KB (98304) - SerializeBatchTime: 37.681us - ThriftTransmitTime(*): 937.72us - UncompressedRowBatchSize: 416.00 B (416) AGGREGATION_NODE (id=1):(Total: 7s341ms, non-child: 6s137ms, % non-child: 83.60%) ExecOption: Codegen Enabled - BuildTime: 5s742ms - GetNewBlockTime: 108.785us - GetResultsTime: 20.432us - HashBuckets: 14.34K (14336) - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 32 - PeakMemoryUsage: 6.31 MB (6619136) - PinTime: 0ns - RowsRepartitioned: 0 - RowsReturned: 20 - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 - UnpinTime: 2.5us HDFS_SCAN_NODE (id=0):(Total: 1s204ms, non-child: 1s204ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 3:16/902.33 MB 4:18/888.90 MB 2:17/924.45 MB 0:19/1.00 GB 1:24/1.17 GB Hdfs Read Thread Concurrency Bucket: 0:85.71% 1:14.29% 2:0% 3:0% 4:0% 5:0% 6:0% File Formats: TEXT/NONE:94 ExecOption: Codegen enabled: 94 out of 94 BytesRead(500.0ms): 0, 704.35 MB, 1005.34 MB, 1.33 GB, 1.68 GB, 2.01 GB, 2.35 GB, 2.70 GB, 2.93 GB, 3.35 GB, 3.59 GB, 3.93 GB, 4.26 GB, 4.57 GB, 4.81 GB - AverageHdfsReadThreadConcurrency: 0.14 - AverageScannerThreadConcurrency: 20.14 - BytesRead: 4.82 GB (5178211951) - BytesReadDataNodeCache: 0 - BytesReadLocal: 4.82 GB (5178198639) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 4.82 GB (5178198639) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumDisksAccessed: 5 - NumScannerThreadsStarted: 23 - PeakMemoryUsage: 628.07 MB (658575360) - PerReadThreadRawHdfsThroughput: 1.57 GB/sec - RemoteScanRanges: 65 - RowsRead: 221.72M (221718312) - RowsReturned: 221.72M (221718312) - RowsReturnedRate: 184.15 M/sec - ScanRangesComplete: 94 - ScannerThreadsInvoluntaryContextSwitches: 7.48K (7479) - ScannerThreadsTotalWallClockTime: 2m22s - DelimiterParseTime: 12s214ms - MaterializeTupleTime(*): 7s790ms - ScannerThreadsSysTime: 1s289ms - ScannerThreadsUserTime: 23s204ms - ScannerThreadsVoluntaryContextSwitches: 241.99K (241988) - TotalRawHdfsReadTime(*): 3s068ms - TotalReadThroughput: 656.31 MB/sec [localhost:21000] >