IGFS and Hadoop accelerator are to be discontinued soon: https://issues.apache.org/jira/browse/IGNITE-11942
Contact the Ignite community for alternate solutions. Some of the solutions will be documented later.
Ignite Hadoop Accelerator contains implementation of
IGFS secondary file system
IgniteHadoopIgfsSecondaryFileSystem which allows read-through and write-through for any Hadoop
To use the secondary file system specify it in
IGFS configuration or in your Java source code:
<bean class="org.apache.ignite.configuration.FileSystemConfiguration"> ... <property name="secondaryFileSystem"> <bean class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem"> <constructor-arg value="hdfs://myHdfs:9000"/> </bean> </property> </bean>
FileSystemConfiguration fileSystemCfg = new FileSystemConfiguration(); ... IgniteHadoopIgfsSecondaryFileSystem hadoopFileSystem = new IgniteHadoopIgfsSecondaryFileSystem("hdfs://myHdfs:9000"); ... fileSystemCfg.setSecondarFileSystem(hadoopFileSystem);
By default, Apache Ignite will not have Hadoop libraries in the classpath during an Apache Ignite node startup. If you decide to use
HDFS as a secondary file system then you have to follow these steps in advance:
Use "Apache Ignite Hadoop Accelerator" edition of Ignite distribution (use -Dignite.edition=hadoop if you're building the distribution by yourself).
Set HADOOP_HOME environment variable before starting an Apache Ignite node if you're using Apache Hadoop distribution. If you use some other Hadoop distribution (HDP, Cloudera, BigTop, etc.) make sure that
/etc/default/hadoopfile exists and has appropriate content.
See respective Ignite installation guide for your Hadoop distribution for details.
Alternatively, you can manually add necessary Hadoop dependencies to Ignite node classpath: these are dependencies of groupId "org.apache.hadoop" listed in file
modules/hadoop/pom.xml. Currently they are the following:
Updated 6 months ago