Package dev.vortex.spark.io

Class VortexIo

java.lang.Object
dev.vortex.spark.io.VortexIo
All Implemented Interfaces:
Serializable
public final class VortexIo extends Object implements Serializable
The Hadoop configuration this Spark job reaches storage with, and the read settings that go with it.

File contents are read and written through Hadoop streams, so the connector sees the same schemes and credential providers as Spark's file index and commit protocol. Reads open a HadoopReadable; writes go through HadoopWritable on the task path Spark's commit protocol assigns.

Built on the driver and shipped to executors, so the configuration travels with it, or is broadcast to them.

See Also:

  • Field Details

    • READ_CONCURRENCY_OPTION

      public static final String READ_CONCURRENCY_OPTION
      Option bounding how many concurrent read upcalls the native reader issues against one file. Each in-flight upcall leases one pooled Hadoop input stream. Zero keeps the native default.
      See Also:

    • COALESCE_DISTANCE_OPTION

      public static final String COALESCE_DISTANCE_OPTION
      Maximum gap between merged reads, in bytes. Defaults to 1 MiB; zero merges only adjacent or overlapping reads.
      See Also:

    • COALESCE_MAX_SIZE_OPTION

      public static final String COALESCE_MAX_SIZE_OPTION
      Maximum span of a merged read, in bytes. Must be positive; defaults to 16 MiB. Individual reads may be larger.
      See Also:

    • SPLIT_CONCURRENCY_OPTION

      public static final String SPLIT_CONCURRENCY_OPTION
      Option bounding how many splits of a file the native scan reads at once, in all. Each split in flight holds the decoded chunks of every column it overlaps, and every task in an executor runs its own scan, which by default keeps several splits in flight per core. Lowering this reduces memory held by concurrent split decoding and Arrow conversion. A scan with both a filter and a limit may also retain one trimmed array per split while preparing output. This does not cap total native memory or I/O read-ahead. Zero keeps the native default.
      See Also:

  • Method Details

    • create

      public static VortexIo create(VortexOptions options, org.apache.hadoop.conf.Configuration hadoopConf)
      Captures hadoopConf and the read settings found in the format options.

    • broadcast

      public static VortexIo broadcast(VortexOptions options, org.apache.hadoop.conf.Configuration hadoopConf, org.apache.spark.SparkContext context)
      Like create(dev.vortex.spark.VortexOptions, org.apache.hadoop.conf.Configuration), but broadcasts hadoopConf through context. Each task that deserializes the result then shares the executor's copy, instead of reading the whole configuration back from its own bytes.

    • defaults

      public static VortexIo defaults()
      Hadoop I/O over a default configuration, for callers that have no Spark session to draw one from.

    • hadoopConf

      public org.apache.hadoop.conf.Configuration hadoopConf()

    • readConcurrency

      public int readConcurrency()
      Bound on concurrent read upcalls per file; zero keeps the native default.

    • splitConcurrency

      public int splitConcurrency()
      The most splits of a file the native scan reads at once; zero keeps the native default.

    • openReadable

      public dev.vortex.io.NativeReadable openReadable(VortexFile file)
      Opens a byte source for file, stating it only if the listing that produced it did not report a size. The caller owns the result and must close it once the scan built on it is done.

      The readers skip listed files without VortexFile.EXTENSION before they get here. The check stops any other path from opening such a file as Vortex data.

      Throws:
      IllegalArgumentException - if the path does not carry VortexFile.EXTENSION