使用windows快捷搭建起一个单机flink
首先找到apache.flink的镜像包官网,选择需要的版本
解压到本地后获得一个flink目录,找到bin目录,可以看到flink的启动文件
但是缺少在windows系统下可以使用的.bat文件,在网上找相关资料,找到如下内容,只需要创建并导入这些文件内容即可:
flink.bat文件如下:
::###############################################################################
:: Licensed to the Apache Software Foundation (ASF) under one
:: or more contributor license agreements. See the NOTICE file
:: distributed with this work for additional information
:: regarding copyright ownership. The ASF licenses this file
:: to you under the Apache License, Version 2.0 (the
:: "License"); you may not use this file except in compliance
:: with the License. You may obtain a copy of the License at
::
:: http://www.apache.org/licenses/LICENSE-2.0
::
:: Unless required by applicable law or agreed to in writing, software
:: distributed under the License is distributed on an "AS IS" BASIS,
:: WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
:: See the License for the specific language governing permissions and
:: limitations under the License.
::###############################################################################
@echo off
setlocal
SET bin=%~dp0
SET FLINK_HOME=%bin%..
SET FLINK_LIB_DIR=%FLINK_HOME%\lib
SET FLINK_PLUGINS_DIR=%FLINK_HOME%\plugins
SET JVM_ARGS=-Xmx512m
SET FLINK_JM_CLASSPATH=%FLINK_LIB_DIR%\*
java %JVM_ARGS% -cp "%FLINK_JM_CLASSPATH%"; org.apache.flink.client.cli.CliFrontend %*
endlocal
start-cluster.bat文件如下:
::###############################################################################
:: Licensed to the Apache Software Foundation (ASF) under one
:: or more contributor license agreements. See the NOTICE file
:: distributed with this work for additional information
:: regarding copyright ownership. The ASF licenses this file
:: to you under the Apache License, Version 2.0 (the
:: "License"); you may not use this file except in compliance
:: with the License. You may obtain a copy of the License at
::
:: http://www.apache.org/licenses/LICENSE-2.0
::
:: Unless required by applicable law or agreed to in writing, software
:: distributed under the License is distributed on an "AS IS" BASIS,
:: WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
:: See the License for the specific language governing permissions and
:: limitations under the License.
::###############################################################################
@echo off
setlocal EnableDelayedExpansion
SET bin=%~dp0
SET FLINK_HOME=%bin%..
SET FLINK_LIB_DIR=%FLINK_HOME%\lib
SET FLINK_PLUGINS_DIR=%FLINK_HOME%\plugins
SET FLINK_CONF_DIR=%FLINK_HOME%\conf
SET FLINK_LOG_DIR=%FLINK_HOME%\log
SET JVM_ARGS=-Xms1024m -Xmx1024m
SET FLINK_CLASSPATH=%FLINK_LIB_DIR%\*
SET logname_jm=flink-%username%-jobmanager.log
SET logname_tm=flink-%username%-taskmanager.log
SET log_jm=%FLINK_LOG_DIR%\%logname_jm%
SET log_tm=%FLINK_LOG_DIR%\%logname_tm%
SET outname_jm=flink-%username%-jobmanager.out
SET outname_tm=flink-%username%-taskmanager.out
SET out_jm=%FLINK_LOG_DIR%\%outname_jm%
SET out_tm=%FLINK_LOG_DIR%\%outname_tm%
SET log_setting_jm=-Dlog.file="%log_jm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties"
SET log_setting_tm=-Dlog.file="%log_tm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties"
:: Log rotation (quick and dirty)
CD "%FLINK_LOG_DIR%"
for /l %%x in (5, -1, 1) do (
SET /A y = %%x+1
RENAME "%logname_jm%.%%x" "%logname_jm%.!y!" 2> nul
RENAME "%logname_tm%.%%x" "%logname_tm%.!y!" 2> nul
RENAME "%outname_jm%.%%x" "%outname_jm%.!y!" 2> nul
RENAME "%outname_tm%.%%x" "%outname_tm%.!y!" 2> nul
)
RENAME "%logname_jm%" "%logname_jm%.0" 2> nul
RENAME "%logname_tm%" "%logname_tm%.0" 2> nul
RENAME "%outname_jm%" "%outname_jm%.0" 2> nul
RENAME "%outname_tm%" "%outname_tm%.0" 2> nul
DEL "%logname_jm%.6" 2> nul
DEL "%logname_tm%.6" 2> nul
DEL "%outname_jm%.6" 2> nul
DEL "%outname_tm%.6" 2> nul
for %%X in (java.exe) do (set FOUND=%%~$PATH:X)
if not defined FOUND (
echo java.exe was not found in PATH variable
goto :eof
)
echo Starting a local cluster with one JobManager process and one TaskManager process.
echo You can terminate the processes via CTRL-C in the spawned shell windows.
echo Web interface by default on http://localhost:8081/.
start /b java %JVM_ARGS% %log_setting_jm% -cp "%FLINK_CLASSPATH%"; org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint --configDir "%FLINK_CONF_DIR%" > "%out_jm%" 2>&1
start /b java %JVM_ARGS% %log_setting_tm% -cp "%FLINK_CLASSPATH%"; org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir "%FLINK_CONF_DIR%" > "%out_tm%" 2>&1
endlocal
然后在cmd中即可启动flink集群,首先进入flink安装目录的bin路径,执行:
start-cluster
--------------------------------------------------------------------------------
Starting a local cluster with one JobManager process and one TaskManager process.
You can terminate the processes via CTRL-C in the spawned shell windows.
Web interface by default on http://localhost:8081/.
看到如下输出,则启动成功,执行jps
命令,看到:
179008 StandaloneSessionClusterEntrypoint
同时在浏览器中输入localhost:8081
即可登录flink管理后台
但是在windows下我没有找到stop-cluster.bat,只能先通过taskkill -f -pid 179008
来完成集群停止了
使用windows+cygwin快捷搭建起一个单机flink
有时开发人员还是更喜欢使用linux系统,不想使用虚拟机环境,也不想开云服务器的情况下,可以使用cygwin搭建
找到cygwin的官方安装地址
下载后打开可执行文件
选择一个合适的根目录,这就是以后cyg虚拟环境使用的root目录,包括linux环境和命令安装都可以从这里进行
补链接图https://blog.csdn.net/weixin_42837669/article/details/114381405
然后到了这一步是要选择需要安装的命令,这里命令非常多,可以只选一些常用的,比如vim、curl等,当然,这里也可以不装命令,先把环境搞起来,后续想要重新安装命令,只需要打开cygwin的安装文件,再次回到这一步,挑选一些命令重新执行安装即可,不会影响当前环境内容
补链接图https://blog.csdn.net/weixin_42837669/article/details/114381405
安装好后即可打开我们的linux环境,这时候我们可以先开启一些命令别名,比如ll命令,在~/.bashrc
文件下,是一个隐藏文件,使用ls -a
命令可以找到,找到:
# alias ll='ls -l'
把这一行的注释解开即可,重启下cygwin的GUI,再打开就可以使用ll命令了
搭建好flink集群后,可以把之前下好的flink目录整体copy到我们自己的路径下,然后还是进入bin目录,这次不需要使用.bat文件了,直接执行以下命令:
./start-cluster.sh
即可启动集群
如果启动没成功,可以去../log
目录下面看看日志,可能有这样的报错:
Improperly specified VM option 'MaxMetaspaceSize=268435456
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
只需要在config.sh里面注释掉如下代码即可:
export JVM_ARGS="${JVM_ARGS} ${jvm_params}"
再次启动就可以了,同样可以使用localhost:8081
即可登录flink管理后台
启动单机flink的examples
可以看到,在flink的expamples目录下面有很多样例jar,选一个有代表性的/examples/streaming/WordCount.jar
执行以下命令:
flink run WordCount.jar
---------------------------
-bash: flink: command not found
解决这个问题的办法有两种,要么去bin目录下面执行./flink run ../examples/streaming/WordCount.jar
要么可以加环境变量解决
环境变量的本质作用是,当执行一个命令,系统先去环境变量所对应的目录下面寻找是否有这个命令
那么我们加以下两个变量:
FLINK_HOME xxxxx\flink-1.15.0\bin
Path xxxxxxxxxxxxx;%FLINK_HOME%
再次执行,如果发现有以下报错:
module java.base does not “opens java.util“ to unnamed module
可能是java版本太高,暂未找到在linux环境下的解决方案,换个jdk8版本再试试
Executing example with default input data.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
Job has been submitted with JobID 7abd1971de905dcc139b55d61333f6db
------------------------------------------------------------
The program finished with the following exception:
……
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not acquire the minimum require
d resources.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
... 37 more
Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not acquire the minimum required resources.
这里又报错了,看上去好像是资源不够,流跑不起来
评论区