diff --git a/README.md b/README.md index c286ba91..03b4bfde 100644 --- a/README.md +++ b/README.md @@ -44,8 +44,8 @@ It is **authored by [Gines Hidalgo](https://www.gineshidalgo.com), [Zhe Cao](htt - **Output**: Basic image + keypoint display/saving (PNG, JPG, AVI, ...), keypoint saving (JSON, XML, YML, ...), and/or keypoints as array class. - **OS**: Ubuntu (14, 16), Windows (8, 10), Mac OSX, Nvidia TX2. - **Training and datasets**: - - [OpenPose Training](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train). - - [Foot dataset website](https://cmu-perceptual-computing-lab.github.io/foot_keypoint_dataset/). + - [**OpenPose Training**](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train). + - [**Foot dataset website**](https://cmu-perceptual-computing-lab.github.io/foot_keypoint_dataset/). - **Others**: - Available: command-line demo, C++ wrapper, and C++ API. - [**Python API**](doc/modules/python_module.md). @@ -55,7 +55,7 @@ It is **authored by [Gines Hidalgo](https://www.gineshidalgo.com), [Zhe Cao](htt ## Latest Features -- Oct 2019: [**Training code released**](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train)! +- Sep 2019: [**Training code released**](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train)! - Jan 2019: [**Unity plugin released**](https://github.com/CMU-Perceptual-Computing-Lab/openpose_unity_plugin)! - Jan 2019: [**Improved Python API**](doc/modules/python_module.md) released! Including body, face, hands, and all the functionality of the C++ API! - Dec 2018: [**Foot dataset released**](https://cmu-perceptual-computing-lab.github.io/foot_keypoint_dataset) and [**new paper released**](https://arxiv.org/abs/1812.08008)! diff --git a/doc/deployment.md b/doc/deployment.md new file mode 100644 index 00000000..d79ffa3b --- /dev/null +++ b/doc/deployment.md @@ -0,0 +1,86 @@ +Deploying OpenPose (Exporting OpenPose to Other Projects) +========================== + +## Contents +1. [Introduction](#introduction) +2. [Third-Party Libraries](#third-party-libraries) +3. [Private OpenPose Include Directory](#private-openpose-include-directory) +4. [Crash and Core Dumped Avoidance](#crash-and-core-dumped-avoidance) +5. [Deploying OpenPose](#deploying-openpose) + 1. [Windows](#windows) + 2. [CMake (Windows, Ubuntu, and Mac)](#cmake-windows-ubuntu-and-mac) + + + +### Introduction +Starting in OpenPose 1.6.0 (GitHub code in or after October 2019), OpenPose has considerable refactor its code to get rid of OpenCV in its headers. This makes OpenPose 1.6 headers different to previous versions and a bit harder to use. However, it allows OpenPose to be exported to other projects without requiring any third-party libraries (except in some special cases detailed below). The greatest benefit of this change: if your project already uses OpenCV, and you add your own version of OpenPose, the OpenCV version used in OpenPose and the one used in your project will not interfere with each other anymore, even if they are different versions! + + + +### Third-Party Libraries +While compiling OpenPose from source, the static library files (`*.a` for Ubuntu, `*.lib` for Windows, etc.) and `include/` directories of all the third-party libraries detailed in [doc/installation.md](./installation.md) are required (GFlags, Glog, OpenCV, Caffe, etc.). However, when deploying OpenPose, fewer dependencies are required: +- GFLags and Glog are required only if the `include/openpose/flags.hpp` file is going to be used (e.g., when intenting to use the command-line interface). +- OpenCV can be optionally included if your project already uses it (but make sure to use the same binaries and include directory of OpenCV for both OpenPose and your project or weird runtime crashes will occur!). Including OpenCV does not increase the functionality of OpenPose, but it makes it easier to use by adding some functions that directly take cv::Mat matrices as input (rather than raw pointers). However, it is optional starting in OpenPose 1.6.0. +- Caffe or any other 3rd-party libraries are not required. + +The static library files (`*.a` for Ubuntu, `*.lib` for Windows, etc.) and `include/` directories are the files that must be included in your project settings. However, the runtime library files (`*.so` for Ubuntu, `*.dll` for Windows, etc.), which are always required, must simply be placed together with the final executable or in default system paths. I.e., these files are only used during runtime, so they do not require any configuration in your project settings. E.g., for Windows, you can simply copy the content of the auto-generated `build/bin/` directory into the path where your executable is located. + + + +### Private OpenPose Include Directory +Inside `include/`, there are 2 directories: `openpose/` and `openpose_private/`. Adding the `include_private` directory will require to include more libraries (e.g., OpenCV and Eigen). This directory exposes some extra functions used internally, but most of the cases this functionality is not required at all, so the `include/` directory should only contain the `openpose/` directory when exported. + +Windows-only: In addition, Windows users have to manually add `OP_API` to all the functions/classes from `openpose_private/` that he desires to use and then re-compile OpenPose. + + + +### Crash and Core Dumped Avoidance +If your project already uses OpenCV, and you add your own version of OpenPose, the OpenCV version of OpenPose and the one from your project will not interfere anymore, even if they are different versions. However, you cannot use the OpenCV functions of OpenPose from a different project if that project uses a different versions of OpenCV. Otherwise, very cryptic runtime DLL errors might occur! Make sure you either: +- Compile OpenPose and your project with the same version of OpenCV. +- Or if that is not possible (new since OpenPose 1.6.0), use the non-OpenCV analog functions of OpenPose to avoid cryptic DLL runtime crashes. + + + +## Deploying OpenPose +### Windows +First of all, make sure to read all the sections above. + +Second, note that the CMake option should also work for Windows. Alternatively, we also show the more Windows-like version in which `*.dll`, `*.lib`, and `include/` files are copied, which might be easier to apply when using the portable binaries. + + + +### CMake (Windows, Ubuntu, and Mac) +First of all, make sure to read all the sections above. + +If you only intend to use the OpenPose demo, you might skip this step. This step is only recommended if you plan to use the OpenPose API from other projects. + +To install the OpenPose headers and libraries into the system environment path (e.g., `/usr/local/` or `/usr/`), run the following command. +``` +cd build/ +sudo make install +``` + +Once the installation is completed, you can use OpenPose in your other project using the `find_package` cmake command. Below, is a small example `CMakeLists.txt`. In order to use this script, you also need to copy `FindGFlags.cmake` and `FindGlog.cmake` into your `/cmake/Modules/` (create the directory if necessary). +``` +cmake_minimum_required(VERSION 2.8.7) + +add_definitions(-std=c++11) + +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules") + +find_package(GFlags) +find_package(Glog) +find_package(OpenCV) +find_package(OpenPose REQUIRED) + +include_directories(${OpenPose_INCLUDE_DIRS} ${GFLAGS_INCLUDE_DIR} ${GLOG_INCLUDE_DIR} ${OpenCV_INCLUDE_DIRS}) + +add_executable(example.bin example.cpp) + +target_link_libraries(example.bin ${OpenPose_LIBS} ${GFLAGS_LIBRARY} ${GLOG_LIBRARY} ${OpenCV_LIBS}) +``` + +If Caffe was built with OpenPose, it will automatically find it. Otherwise, you will need to link Caffe again as shown below (otherwise, you might get an error like `/usr/bin/ld: cannot find -lcaffe`). +``` +link_directories(/caffe/build/install/lib) +``` diff --git a/doc/installation.md b/doc/installation.md index 5188770b..3315d473 100644 --- a/doc/installation.md +++ b/doc/installation.md @@ -11,7 +11,8 @@ OpenPose - Installation 7. [Installation](#installation) 8. [Reinstallation](#reinstallation) 9. [Uninstallation](#uninstallation) -10. [Optional Settings](#optional-settings) +10. [Deploying OpenPose (Exporting OpenPose to Other Projects)](#desploying-openpose-exporting-openpose-to-other-projects) +11. [Optional Settings](#optional-settings) 1. [Maximum Speed](#maximum-speed) 2. [COCO and MPI Models](#coco-and-mpi-models) 3. [Python API](#python-api) @@ -129,7 +130,6 @@ The instructions in this section describe the steps to build OpenPose using CMak 3. [OpenPose Configuration](#openpose-configuration) 4. [OpenPose Building](#openpose-building) 5. [Run OpenPose](#run-openpose) -6. [OpenPose from other Projects (Ubuntu and Mac)](#openpose-from-other-projects-ubuntu-and-mac) @@ -207,42 +207,6 @@ Check OpenPose was properly installed by running it on the default images, video -### OpenPose from other Projects (Ubuntu and Mac) -If you only intend to use the OpenPose demo, you might skip this step. This step is only recommended if you plan to use the OpenPose API from other projects. - -To install the OpenPose headers and libraries into the system environment path (e.g., `/usr/local/` or `/usr/`), run the following command. -``` -cd build/ -sudo make install -``` - -Once the installation is completed, you can use OpenPose in your other project using the `find_package` cmake command. Below, is a small example `CMakeLists.txt`. In order to use this script, you also need to copy `FindGFlags.cmake` and `FindGlog.cmake` into your `/cmake/Modules/` (create the directory if necessary). -``` -cmake_minimum_required(VERSION 2.8.7) - -add_definitions(-std=c++11) - -list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules") - -find_package(GFlags) -find_package(Glog) -find_package(OpenCV) -find_package(OpenPose REQUIRED) - -include_directories(${OpenPose_INCLUDE_DIRS} ${GFLAGS_INCLUDE_DIR} ${GLOG_INCLUDE_DIR} ${OpenCV_INCLUDE_DIRS}) - -add_executable(example.bin example.cpp) - -target_link_libraries(example.bin ${OpenPose_LIBS} ${GFLAGS_LIBRARY} ${GLOG_LIBRARY} ${OpenCV_LIBS}) -``` - -If Caffe was built with OpenPose, it will automatically find it. Otherwise, you will need to link Caffe again as shown below (otherwise, you might get an error like `/usr/bin/ld: cannot find -lcaffe`). -``` -link_directories(/caffe/build/install/lib) -``` - - - ## Reinstallation In order to re-install OpenPose: 1. (Ubuntu and Mac) If you ran `sudo make install`, then run `sudo make uninstall` in `build/`. @@ -259,6 +223,11 @@ In order to uninstall OpenPose: +## Deploying OpenPose (Exporting OpenPose to Other Projects) +See [doc/deployment.md](./deployment.md). + + + ### Optional Settings #### Maximum Speed Check the OpenPose Benchmark as well as some hints to speed up and/or reduce the memory requirements for OpenPose on [doc/speed_up_openpose.md](./speed_up_openpose.md). diff --git a/doc/release_notes.md b/doc/release_notes.md index 8950e449..6411d467 100644 --- a/doc/release_notes.md +++ b/doc/release_notes.md @@ -125,7 +125,7 @@ OpenPose Library - Release Notes 3. CvMatToOutput and Renderers allow to keep input resolution as output for images (core module). 6. New standalone face keypoint detector based on OpenCV face detector: much faster if body keypoint detection is not required but much less accurate. 7. Face and hand keypoint detectors now can return each keypoint heatmap. - 8. The flag `USE_CUDNN` is no longer required; `USE_CAFFE` and `USE_CUDA` (replacing the old `CPU_ONLY`) are no longer required to use the library, only to build it. In addition, Boost, Caffe, and its dependencies have been removed from the OpenPose header files. Only OpenCV include and lib folders are required when building a project using OpenPose. + 8. The flag `USE_CUDNN` is no longer required; `USE_CAFFE` and `USE_CUDA` (replacing the old `CPU_ONLY`) are no longer required to use the library, only to build it. In addition, Boost, Caffe, and its dependencies have been removed from the OpenPose header files. Only OpenCV include and lib directories are required when building a project using OpenPose. 9. OpenPose successfully compiles if the flags `USE_CAFFE` and/or `USE_CUDA` are not enabled, although it will give an error saying they are required. 10. COCO JSON file outputs 0 as score for non-detected keypoints. 11. Added example for OpenPose for user asynchronous output and cleaned all `tutorial_wrapper/` examples. @@ -247,7 +247,7 @@ OpenPose Library - Release Notes 1. Removed scale parameter from hand and face rectangle extractor (causing wrong results if custom `--output_resolution`). 2. Functions `scaleKeypoints`, other than `scaleKeypoints(Array& keypoints, const float scale)`, renamed as `scaleKeypoints2d`. 3. `(W)PoseExtractor` renamed to `(W)PoseExtractorNet` to distinguish from new `PoseExtractor`. Analogously with `(W)FaceExtractorNet` and `(W)HandExtractorNet`. - 4. Experimental module removed and internal `tracking` folder moved to main openpose folder. + 4. Experimental module removed and internal `tracking` directory moved to main openpose directory. 5. Switched GUI shortcuts for the kind of channel to render (skeleton, heatmap, PAF, ...) in order to make it more intuitive: 1 for skeleton, 1 for background heatmap, 2 for adding all heatmaps, 3 for adding all PAFs, and 4 to 0 for the initial heatmaps. 3. Main bugs fixed: 1. Fixed hand and face extraction and rendering scaling issues when `--output_resolution` is not the default one. @@ -276,7 +276,7 @@ OpenPose Library - Release Notes 2. Renamed `tutorial_wrapper` as `tutorial_api_cpp` as well as new examples were added. 2. Renamed `tutorial_python` as `tutorial_api_python` as well as new examples were added. 3. Renamed `tutorial_thread` as `tutorial_api_thread`, focused in the multi-thread mechanism. - 4. Removed `tutorial_pose`, the folder `tutorial_api_cpp` includes much cleaner and commented examples. + 4. Removed `tutorial_pose`, the directory `tutorial_api_cpp` includes much cleaner and commented examples. 5. Examples do not end in core dumped if an OpenPose exception occurred during initialization, but they are rather closed returning -1. However, it will still results in core dumped if the exception occurs during multi-threading execution. 6. Added new examples, including examples to extract face and/or hand from images. 7. Added `--no_display` flag for the examples that does not use OpenPose output. @@ -345,9 +345,9 @@ OpenPose Library - Release Notes 1. Replaced `--camera_fps` flag by `--write_video_fps`, given that it was a confusing name: It did not affect the webcam FPS, but only the FPS of the output video. In addition, default value changed from 30 to -1. 2. Flag `--hand_tracking` is a subcase of `--hand_detector`, so it has been removed and incorporated as `--hand_detector 3`. 8. Renamed `--frame_keep_distortion` as `--frame_undistort`, which performs the opposite operation (the default value has been also changed to the opposite). - 9. Renamed `--camera_parameter_folder` as `--camera_parameter_path` because it could also take a whole XML file path rather than its parent folder. + 9. Renamed `--camera_parameter_folder` as `--camera_parameter_path` because it could also take a whole XML file path rather than its parent directory. 10. Default value of flag `--scale_gap` changed from 0.3 to 0.25. - 11. Moved most sh scripts into the `scripts/` folder. Only models/getModels.sh and the `*.bat` files are kept under `models/` and `3rdparty/windows`. + 11. Moved most sh scripts into the `scripts/` directory. Only models/getModels.sh and the `*.bat` files are kept under `models/` and `3rdparty/windows`. 12. For Python compatibility and scalability increase, template `TDatums` used for `include/openpose/wrapper/wrapper.hpp` has changed from `std::vector` to `std::vector>`, including the respective changes in all the worker classes. In addition, some template classes have been simplified to only take 1 template parameter for user simplicity. 13. Renamed intRound, charRound, etc. by positiveIntRound, positiveCharRound, etc. so that people can realize it is not safe for negative numbers. 14. Replaced flag `--write_coco_foot_json` by `--write_coco_json_variants` in order to generalize to any COCO JSON format (i.e., hand, face, etc). @@ -357,7 +357,7 @@ OpenPose Library - Release Notes 3. Template functions could not be imported in Windows for projects using the OpenPose library DLL. 4. Function `scaleKeypoints2d` was not working if any of the scales was 1 (e.g., fail if scaleX = 1 but scaleY != 1, or if any offset was not 0). 5. Fixed bug in `KeepTopNPeople` that could provoke segmentation fault for `number_people_max` > 1. - 6. Camera parameter reader can now take folder paths even if they are not finished in `/` (e.g., `~/Desktop/` worked but `~/Desktop` did not). + 6. Camera parameter reader can now take directory paths even if they are not finished in `/` (e.g., `~/Desktop/` worked but `~/Desktop` did not). 7. 3D module: If the image area was smaller than HD resolution image area, the 3D keypoints were not properly estimated. 8. OpenCL fixes. 9. If manual CUDA architectures are set in CMake, they are also set for Caffe rather than only for OpenPose. @@ -383,10 +383,13 @@ OpenPose Library - Release Notes -## Current version - Future OpenPose 1.5.2 +## Current version - Future OpenPose 1.6.0 1. Main improvements: - 1. Default OpenCV version for Windows upgraded to version 4.1.1, extracted from their oficial website: section `Releases`, subsection `OpenCV – 4.1.1`, `Windows` version. + 1. Headers do not contain any 3rd-party library includes nor functions. This way, OpenPose can be exported without needing 3rd-party includes nor static library files (e.g., lib files in Windows), allowing people to use their own versions of OpenCV, Eigen, etc. without conflicting with OpenPose. Dynamic library files (e.g., `dll` files in Windows, `so` in Ubuntu) are still required. + 2. Created the `openpose_private` directory with some internal headers that, if exported with OpenPose, would require including 3rd-party headers and static library files. + 3. Default OpenCV version for Windows upgraded to version 4.1.1, extracted from their oficial website: section `Releases`, subsection `OpenCV - 4.1.1`, `Windows` version. 2. Functions or parameters renamed: + 1. All headers moved into `openpose_private` and all 3rd-party library calls in headers. 3. Main bugs fixed: 1. Removed many Visual Studio (Windows) warnings. 4. Changes/additions that affect the compatibility with the OpenPose Unity Plugin: diff --git a/doc/released_features.md b/doc/released_features.md index 2133c6ac..cd5c3b13 100644 --- a/doc/released_features.md +++ b/doc/released_features.md @@ -1,7 +1,7 @@ OpenPose Library - All Released Features ==================================== -- Oct 2019: [**Training code released**](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train)! +- Sep 2019: [**Training code released**](https://github.com/CMU-Perceptual-Computing-Lab/openpose_train)! - Jan 2019: [**Unity plugin released**](https://github.com/CMU-Perceptual-Computing-Lab/openpose_unity_plugin)! - Jan 2019: [**Improved Python API**](doc/modules/python_module.md) released! Including body, face, hands, and all the functionality of the C++ API! - Dec 2018: [**Foot dataset released**](https://cmu-perceptual-computing-lab.github.io/foot_keypoint_dataset) and [**new paper released**](https://arxiv.org/abs/1812.08008)! diff --git a/examples/calibration/calibration.cpp b/examples/calibration/calibration.cpp index 3ce4d51b..2f70bba3 100644 --- a/examples/calibration/calibration.cpp +++ b/examples/calibration/calibration.cpp @@ -3,7 +3,9 @@ // Implemented on top of OpenCV. // It computes and saves the intrinsics parameters of the input images. -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_POSE #include // OpenPose dependencies @@ -47,6 +49,7 @@ int openPoseDemo() // Common parameters const auto gridInnerCorners = op::flagsToPoint(FLAGS_grid_number_inner_corners, "12x7"); const auto calibrationImageDir = op::formatAsDirectory(FLAGS_calibration_image_dir); + const auto gridSqureSizeMm = (float)FLAGS_grid_square_size_mm; // Calibration - Intrinsics if (FLAGS_mode == 1) @@ -61,7 +64,7 @@ int openPoseDemo() const auto saveImagesWithCorners = true; // Run calibration op::estimateAndSaveIntrinsics( - gridInnerCorners, FLAGS_grid_square_size_mm, flags, + gridInnerCorners, gridSqureSizeMm, flags, op::formatAsDirectory(FLAGS_camera_parameter_folder), calibrationImageDir, FLAGS_camera_serial_number, saveImagesWithCorners); op::log("Intrinsic calibration completed!", op::Priority::High); @@ -73,7 +76,7 @@ int openPoseDemo() op::log("Running calibration (extrinsic parameters)...", op::Priority::High); // Run calibration op::estimateAndSaveExtrinsics( - FLAGS_camera_parameter_folder, calibrationImageDir, gridInnerCorners, FLAGS_grid_square_size_mm, + FLAGS_camera_parameter_folder, calibrationImageDir, gridInnerCorners, gridSqureSizeMm, FLAGS_cam0, FLAGS_cam1, FLAGS_omit_distortion, FLAGS_combine_cam0_extrinsics); // Logging op::log("Extrinsic calibration completed!", op::Priority::High); @@ -91,7 +94,7 @@ int openPoseDemo() const auto saveImagesWithCorners = false; // Run calibration op::refineAndSaveExtrinsics( - FLAGS_camera_parameter_folder, calibrationImageDir, gridInnerCorners, FLAGS_grid_square_size_mm, + FLAGS_camera_parameter_folder, calibrationImageDir, gridInnerCorners, gridSqureSizeMm, FLAGS_number_cameras, FLAGS_omit_distortion, saveImagesWithCorners); // Logging op::log("Extrinsic calibration (bundle adjustment) completed!", op::Priority::High); @@ -120,7 +123,7 @@ int openPoseDemo() return 0; } - catch (const std::exception& e) + catch (const std::exception&) { return -1; } diff --git a/examples/openpose/openpose.cpp b/examples/openpose/openpose.cpp index 290d76d3..cea81677 100755 --- a/examples/openpose/openpose.cpp +++ b/examples/openpose/openpose.cpp @@ -7,7 +7,7 @@ // If the user wants to learn to use the OpenPose C++ library, we highly recommend to start with the examples in // `examples/tutorial_api_cpp/`. -// Command-line user intraface +// Command-line user interface #include // OpenPose dependencies #include diff --git a/examples/tests/clTest.cpp b/examples/tests/clTest.cpp index 4f1391ec..2104a4f7 100644 --- a/examples/tests/clTest.cpp +++ b/examples/tests/clTest.cpp @@ -1,6 +1,8 @@ // ------------------------- OpenPose Resize Layer Testing ------------------------- -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_POSE #include // OpenPose dependencies @@ -11,8 +13,8 @@ #endif // OpenCL dependencies #ifdef USE_OPENCL -#include -#include +#include +#include DEFINE_string(image_path, "examples/media/COCO_val2014_000000000192.jpg", "Process the desired image."); diff --git a/examples/tests/handFromJsonTest.cpp b/examples/tests/handFromJsonTest.cpp index fe12efec..172671b8 100644 --- a/examples/tests/handFromJsonTest.cpp +++ b/examples/tests/handFromJsonTest.cpp @@ -1,7 +1,7 @@ // ----------------------- OpenPose Tests - Hand Keypoint Detection from JSON Ground-Truth Data ----------------------- // Example to test hands accuracy given ground-truth bounding boxes. -// Command-line user intraface +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_POSE #include // OpenPose dependencies diff --git a/examples/tests/resizeTest.cpp b/examples/tests/resizeTest.cpp index 3eef40ed..e50b839c 100644 --- a/examples/tests/resizeTest.cpp +++ b/examples/tests/resizeTest.cpp @@ -1,6 +1,8 @@ // ------------------------- OpenPose Resize Layer Testing ------------------------- -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_POSE #include // OpenPose dependencies @@ -77,7 +79,8 @@ try { // logging_level - cv::Mat img = op::loadImage(FLAGS_image_path, CV_LOAD_IMAGE_GRAYSCALE); + op::Matrix opImg = op::loadImage(FLAGS_image_path, CV_LOAD_IMAGE_GRAYSCALE); + cv::Mat img = OP_OP2CVMAT(opImg); if(img.empty()) op::error("Could not open or find the image: " + FLAGS_image_path, __LINE__, __FUNCTION__, __FILE__); img.convertTo(img, CV_32FC1); diff --git a/examples/tests/wrapperHandFromJsonTest.hpp b/examples/tests/wrapperHandFromJsonTest.hpp index d94d6247..d58d2d40 100644 --- a/examples/tests/wrapperHandFromJsonTest.hpp +++ b/examples/tests/wrapperHandFromJsonTest.hpp @@ -1,6 +1,9 @@ #ifndef OPENPOSE_WRAPPER_WRAPPER_HAND_FROM_JSON_TEST_HPP #define OPENPOSE_WRAPPER_WRAPPER_HAND_FROM_JSON_TEST_HPP +// Third-party dependencies +#include +// OpenPose dependencies #include namespace op @@ -59,7 +62,7 @@ namespace op /** * Set ThreadManager from TWorkers (private internal function). - * After any configure() has been called, the TWorkers are initialized. This function resets the ThreadManager and adds them. + * After any configure() has been called, the TWorkers are initialized. This function resets the ThreadManager and adds them. * Common code for start() and exec(). */ void configureThreadManager(); @@ -292,7 +295,7 @@ namespace op try { mThreadManager.reset(); - // Reset + // Reset wDatumProducer = nullptr; spWScaleAndSizeExtractor = nullptr; spWCvMatToOpInput = nullptr; diff --git a/examples/tutorial_add_module/1_custom_post_processing.cpp b/examples/tutorial_add_module/1_custom_post_processing.cpp index 30c42480..fae1d14b 100644 --- a/examples/tutorial_add_module/1_custom_post_processing.cpp +++ b/examples/tutorial_add_module/1_custom_post_processing.cpp @@ -23,7 +23,7 @@ // This example is a sub-case of `tutorial_api_cpp/15_synchronous_custom_postprocessing.cpp`, where only custom post-processing is // considered. -// Command-line user intraface +// Command-line user interface #include // OpenPose dependencies #include diff --git a/examples/tutorial_add_module/userDatum.hpp b/examples/tutorial_add_module/userDatum.hpp index e5857e64..903df0bb 100644 --- a/examples/tutorial_add_module/userDatum.hpp +++ b/examples/tutorial_add_module/userDatum.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_TUTORIAL_USER_DATUM_HPP #define OPENPOSE_TUTORIAL_USER_DATUM_HPP -#include // cv::Mat #include #include diff --git a/examples/tutorial_add_module/userPostProcessing.hpp b/examples/tutorial_add_module/userPostProcessing.hpp index befabeab..7dae61f2 100644 --- a/examples/tutorial_add_module/userPostProcessing.hpp +++ b/examples/tutorial_add_module/userPostProcessing.hpp @@ -1,6 +1,9 @@ #ifndef OPENPOSE_EXAMPLES_TUTORIAL_USER_POST_PROCESSING_HPP #define OPENPOSE_EXAMPLES_TUTORIAL_USER_POST_PROCESSING_HPP +// Third-party dependencies +#include +// OpenPose dependencies #include #include diff --git a/examples/tutorial_add_module/wUserPostProcessing.hpp b/examples/tutorial_add_module/wUserPostProcessing.hpp index 9c32e792..bdb3ca42 100644 --- a/examples/tutorial_add_module/wUserPostProcessing.hpp +++ b/examples/tutorial_add_module/wUserPostProcessing.hpp @@ -56,9 +56,12 @@ namespace op // Profiling speed const auto profilerKey = Profiler::timerInit(__LINE__, __FUNCTION__, __FILE__); for (auto& datum : *tDatums) - // THIS IS THE ONLY LINE THAT THE USER MUST MODIFY ON THIS HPP FILE, by using the proper function - // and datum elements - spUserPostProcessing->doSomething(datum->cvOutputData, datum->cvOutputData); + { + // THESE 2 ARE THE ONLY LINES THAT THE USER MUST MODIFY ON THIS HPP FILE, by using the proper + // function and datum elements + cv::Mat cvOutputData = OP_OP2CVMAT(datum->cvOutputData); + spUserPostProcessing->doSomething(cvOutputData, cvOutputData); + } // Profiling speed Profiler::timerEnd(profilerKey); Profiler::printAveragedTimeMsOnIterationX(profilerKey, __LINE__, __FUNCTION__, __FILE__); diff --git a/examples/tutorial_api_cpp/01_body_from_image_default.cpp b/examples/tutorial_api_cpp/01_body_from_image_default.cpp index d5977dcc..52ff6f3e 100644 --- a/examples/tutorial_api_cpp/01_body_from_image_default.cpp +++ b/examples/tutorial_api_cpp/01_body_from_image_default.cpp @@ -1,7 +1,9 @@ // ----------------------------- OpenPose C++ API Tutorial - Example 1 - Body from image ----------------------------- // It reads an image, process it, and displays it with the pose keypoints. -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_POSE #include // OpenPose dependencies @@ -26,7 +28,8 @@ void display(const std::shared_ptr>>& dat if (datumsPtr != nullptr && !datumsPtr->empty()) { // Display image - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); cv::waitKey(0); } else @@ -99,7 +102,8 @@ int tutorialApiCpp() opWrapper.start(); // Process and display image - const auto imageToProcess = cv::imread(FLAGS_image_path); + const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); + const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); auto datumProcessed = opWrapper.emplaceAndPop(imageToProcess); if (datumProcessed != nullptr) { diff --git a/examples/tutorial_api_cpp/02_whole_body_from_image_default.cpp b/examples/tutorial_api_cpp/02_whole_body_from_image_default.cpp index a6ad7adc..98f5e7db 100644 --- a/examples/tutorial_api_cpp/02_whole_body_from_image_default.cpp +++ b/examples/tutorial_api_cpp/02_whole_body_from_image_default.cpp @@ -1,7 +1,9 @@ // -------------------------- OpenPose C++ API Tutorial - Example 2 - Whole body from image -------------------------- // It reads an image, process it, and displays it with the pose, hand, and face keypoints. -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_POSE #include // OpenPose dependencies @@ -26,7 +28,8 @@ void display(const std::shared_ptr>>& dat if (datumsPtr != nullptr && !datumsPtr->empty()) { // Display image - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); cv::waitKey(0); } else @@ -81,7 +84,8 @@ int tutorialApiCpp() opWrapper.start(); // Process and display image - const auto imageToProcess = cv::imread(FLAGS_image_path); + const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); + const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); auto datumProcessed = opWrapper.emplaceAndPop(imageToProcess); if (datumProcessed != nullptr) { diff --git a/examples/tutorial_api_cpp/03_keypoints_from_image.cpp b/examples/tutorial_api_cpp/03_keypoints_from_image.cpp index 940d12fe..42a08365 100644 --- a/examples/tutorial_api_cpp/03_keypoints_from_image.cpp +++ b/examples/tutorial_api_cpp/03_keypoints_from_image.cpp @@ -2,7 +2,9 @@ // It reads an image, process it, and displays it with the pose (and optionally hand and face) keypoints. In addition, // it includes all the OpenPose configuration flags (enable/disable hand, face, output saving, etc.). -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include @@ -28,7 +30,8 @@ void display(const std::shared_ptr>>& dat if (datumsPtr != nullptr && !datumsPtr->empty()) { // Display image - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); cv::waitKey(0); } else @@ -165,7 +168,8 @@ int tutorialApiCpp() opWrapper.start(); // Process and display image - const auto imageToProcess = cv::imread(FLAGS_image_path); + const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); + const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); auto datumProcessed = opWrapper.emplaceAndPop(imageToProcess); if (datumProcessed != nullptr) { diff --git a/examples/tutorial_api_cpp/04_keypoints_from_images.cpp b/examples/tutorial_api_cpp/04_keypoints_from_images.cpp index ab8f3800..a5830135 100644 --- a/examples/tutorial_api_cpp/04_keypoints_from_images.cpp +++ b/examples/tutorial_api_cpp/04_keypoints_from_images.cpp @@ -2,7 +2,9 @@ // It reads images, process them, and display them with the pose (and optionally hand and face) keypoints. In addition, // it includes all the OpenPose configuration flags (enable/disable hand, face, output saving, etc.). -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include @@ -28,7 +30,8 @@ bool display(const std::shared_ptr>>& dat if (datumsPtr != nullptr && !datumsPtr->empty()) { // Display image and sleeps at least 1 ms (it usually sleeps ~5-10 msec to display the image) - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); } else op::log("Nullptr or empty datumsPtr found.", op::Priority::High); @@ -172,7 +175,8 @@ int tutorialApiCpp() // Process and display images for (const auto& imagePath : imagePaths) { - const auto imageToProcess = cv::imread(imagePath); + const cv::Mat cvImageToProcess = cv::imread(imagePath); + const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); auto datumProcessed = opWrapper.emplaceAndPop(imageToProcess); if (datumProcessed != nullptr) { diff --git a/examples/tutorial_api_cpp/05_keypoints_from_images_multi_gpu.cpp b/examples/tutorial_api_cpp/05_keypoints_from_images_multi_gpu.cpp index 0d3295b0..a42b0876 100644 --- a/examples/tutorial_api_cpp/05_keypoints_from_images_multi_gpu.cpp +++ b/examples/tutorial_api_cpp/05_keypoints_from_images_multi_gpu.cpp @@ -2,7 +2,9 @@ // It reads images, process them, and display them with the pose (and optionally hand and face) keypoints. In addition, // it includes all the OpenPose configuration flags (enable/disable hand, face, output saving, etc.). -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include @@ -34,7 +36,8 @@ bool display(const std::shared_ptr>>& dat if (datumsPtr != nullptr && !datumsPtr->empty()) { // Display image and sleeps at least 1 ms (it usually sleeps ~5-10 msec to display the image) - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); } else op::log("Nullptr or empty datumsPtr found.", op::Priority::High); @@ -199,7 +202,8 @@ int tutorialApiCpp() { const auto& imagePath = imagePaths.at(imageId); // Faster alternative that moves imageToProcess - auto imageToProcess = cv::imread(imagePath); + cv::Mat cvImageToProcess = cv::imread(imagePath); + op::Matrix imageToProcess = OP_CV2OPMAT(cvImageToProcess); opWrapper.waitAndEmplace(imageToProcess); // // Slower but safer alternative that copies imageToProcess // const auto imageToProcess = cv::imread(imagePath); @@ -245,7 +249,8 @@ int tutorialApiCpp() for (const auto& imagePath : imagePaths) { // Faster alternative that moves imageToProcess - auto imageToProcess = cv::imread(imagePath); + cv::Mat cvImageToProcess = cv::imread(imagePath); + op::Matrix imageToProcess = OP_CV2OPMAT(cvImageToProcess); opWrapper.waitAndEmplace(imageToProcess); // // Slower but safer alternative that copies imageToProcess // const auto imageToProcess = cv::imread(imagePath); diff --git a/examples/tutorial_api_cpp/06_face_from_image.cpp b/examples/tutorial_api_cpp/06_face_from_image.cpp index a1db25ed..0683d471 100644 --- a/examples/tutorial_api_cpp/06_face_from_image.cpp +++ b/examples/tutorial_api_cpp/06_face_from_image.cpp @@ -5,7 +5,9 @@ // Output: OpenPose face keypoint detection. // NOTE: This demo is auto-selecting the following flags: `--body 0 --face --face_detector 2` -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include @@ -31,7 +33,8 @@ void display(const std::shared_ptr>>& dat if (datumsPtr != nullptr && !datumsPtr->empty()) { // Display image - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); cv::waitKey(0); } else @@ -173,7 +176,8 @@ int tutorialApiCpp() opWrapper.start(); // Read image and face rectangle locations - const auto imageToProcess = cv::imread(FLAGS_image_path); + const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); + const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); const std::vector> faceRectangles{ op::Rectangle{330.119385f, 277.532715f, 48.717274f, 48.717274f}, // Face of person 0 op::Rectangle{24.036991f, 267.918793f, 65.175171f, 65.175171f}, // Face of person 1 diff --git a/examples/tutorial_api_cpp/07_hand_from_image.cpp b/examples/tutorial_api_cpp/07_hand_from_image.cpp index 791fe065..2029a065 100644 --- a/examples/tutorial_api_cpp/07_hand_from_image.cpp +++ b/examples/tutorial_api_cpp/07_hand_from_image.cpp @@ -5,7 +5,9 @@ // Output: OpenPose hand keypoint detection. // NOTE: This demo is auto-selecting the following flags: `--body 0 --hand --hand_detector 2` -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include @@ -31,7 +33,8 @@ void display(const std::shared_ptr>>& dat if (datumsPtr != nullptr && !datumsPtr->empty()) { // Display image - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); cv::waitKey(0); } else @@ -173,7 +176,8 @@ int tutorialApiCpp() opWrapper.start(); // Read image and hand rectangle locations - const auto imageToProcess = cv::imread(FLAGS_image_path); + const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); + const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); const std::vector, 2>> handRectangles{ // Left/Right hands of person 0 std::array, 2>{ diff --git a/examples/tutorial_api_cpp/08_heatmaps_from_image.cpp b/examples/tutorial_api_cpp/08_heatmaps_from_image.cpp index 9374e411..e1cbeab3 100644 --- a/examples/tutorial_api_cpp/08_heatmaps_from_image.cpp +++ b/examples/tutorial_api_cpp/08_heatmaps_from_image.cpp @@ -2,7 +2,9 @@ // It reads an image, process it, and displays it with the body heatmaps. In addition, it includes all the // OpenPose configuration flags (enable/disable hand, face, output saving, etc.). -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include @@ -203,7 +205,8 @@ int tutorialApiCpp() opWrapper.start(); // Process and display image - const auto imageToProcess = cv::imread(FLAGS_image_path); + const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); + const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); auto datumProcessed = opWrapper.emplaceAndPop(imageToProcess); if (datumProcessed != nullptr) { diff --git a/examples/tutorial_api_cpp/09_keypoints_from_heatmaps.cpp b/examples/tutorial_api_cpp/09_keypoints_from_heatmaps.cpp index 2220b32e..cfad04cf 100644 --- a/examples/tutorial_api_cpp/09_keypoints_from_heatmaps.cpp +++ b/examples/tutorial_api_cpp/09_keypoints_from_heatmaps.cpp @@ -5,7 +5,9 @@ // its internal network, or it will lead to core dumped (segmentation) errors. You can modify the pose // estimation flags to match the dimension of both elements (e.g., `--net_resolution`, `--scale_number`, etc.). -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include @@ -31,7 +33,8 @@ void display(const std::shared_ptr>>& dat if (datumsPtr != nullptr && !datumsPtr->empty()) { // Display image - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); cv::waitKey(0); } else @@ -159,7 +162,8 @@ int tutorialApiCpp() const auto opTimer = op::getTimerInit(); // Image to process - const auto imageToProcess = cv::imread(FLAGS_image_path); + const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); + const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); // Required flags to disable the OpenPose network FLAGS_body = 2; diff --git a/examples/tutorial_api_cpp/10_asynchronous_custom_input.cpp b/examples/tutorial_api_cpp/10_asynchronous_custom_input.cpp index c44d0260..cf3d4fc2 100644 --- a/examples/tutorial_api_cpp/10_asynchronous_custom_input.cpp +++ b/examples/tutorial_api_cpp/10_asynchronous_custom_input.cpp @@ -3,7 +3,9 @@ // In this function, the user can implement its own way to create frames (e.g., reading his own folder of images) // and emplaces/pushes the frames to OpenPose. -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #include // OpenPose dependencies @@ -49,7 +51,8 @@ public: datumPtr = std::make_shared(); // Fill datum - datumPtr->cvInputData = cv::imread(mImageFiles.at(mCounter++)); + const cv::Mat cvInputData = cv::imread(mImageFiles.at(mCounter++)); + datumPtr->cvInputData = OP_CV2OPCONSTMAT(cvInputData); // If empty frame -> return nullptr if (datumPtr->cvInputData.empty()) diff --git a/examples/tutorial_api_cpp/11_asynchronous_custom_output.cpp b/examples/tutorial_api_cpp/11_asynchronous_custom_output.cpp index 944b45c3..1a0d53da 100644 --- a/examples/tutorial_api_cpp/11_asynchronous_custom_output.cpp +++ b/examples/tutorial_api_cpp/11_asynchronous_custom_output.cpp @@ -2,7 +2,9 @@ // Asynchronous mode: ideal for fast prototyping when performance is not an issue. // In this function, the user can implement its own way to render/display/storage the results. -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include // OpenPose dependencies @@ -24,8 +26,9 @@ public: // datumPtr->poseKeypoints: Array with the estimated pose if (datumsPtr != nullptr && !datumsPtr->empty()) { - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); // Display image and sleeps at least 1 ms (it usually sleeps ~5-10 msec to display the image) + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); } else op::log("Nullptr or empty datumsPtr found.", op::Priority::High); diff --git a/examples/tutorial_api_cpp/12_asynchronous_custom_input_output_and_datum.cpp b/examples/tutorial_api_cpp/12_asynchronous_custom_input_output_and_datum.cpp index cbc167ee..cd71cd02 100644 --- a/examples/tutorial_api_cpp/12_asynchronous_custom_input_output_and_datum.cpp +++ b/examples/tutorial_api_cpp/12_asynchronous_custom_input_output_and_datum.cpp @@ -3,7 +3,9 @@ // In this function, the user can implement its own way to create frames (e.g., reading his own folder of images) // and its own way to render/display them after being processed by OpenPose. -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include @@ -66,7 +68,8 @@ public: datumPtr = std::make_shared(); // Fill datum - datumPtr->cvInputData = cv::imread(mImageFiles.at(mCounter++)); + const cv::Mat cvInputData = cv::imread(mImageFiles.at(mCounter++)); + datumPtr->cvInputData = OP_CV2OPCONSTMAT(cvInputData); // If empty frame -> return nullptr if (datumPtr->cvInputData.empty()) @@ -106,7 +109,8 @@ public: if (datumsPtr != nullptr && !datumsPtr->empty()) { // Display image and sleeps at least 1 ms (it usually sleeps ~5-10 msec to display the image) - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); } else op::log("Nullptr or empty datumsPtr found.", op::Priority::High); diff --git a/examples/tutorial_api_cpp/13_synchronous_custom_input.cpp b/examples/tutorial_api_cpp/13_synchronous_custom_input.cpp index 7eb99eec..397c1593 100644 --- a/examples/tutorial_api_cpp/13_synchronous_custom_input.cpp +++ b/examples/tutorial_api_cpp/13_synchronous_custom_input.cpp @@ -3,7 +3,9 @@ // performance. // In this function, the user can implement its own way to create frames (e.g., reading his own folder of images). -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #include // OpenPose dependencies @@ -53,7 +55,8 @@ public: datumPtr = std::make_shared(); // Fill datum - datumPtr->cvInputData = cv::imread(mImageFiles.at(mCounter++)); + const cv::Mat cvInputData = cv::imread(mImageFiles.at(mCounter++)); + datumPtr->cvInputData = OP_CV2OPCONSTMAT(cvInputData); // If empty frame -> return nullptr if (datumPtr->cvInputData.empty()) diff --git a/examples/tutorial_api_cpp/14_synchronous_custom_preprocessing.cpp b/examples/tutorial_api_cpp/14_synchronous_custom_preprocessing.cpp index 750e0b6c..95ec4292 100644 --- a/examples/tutorial_api_cpp/14_synchronous_custom_preprocessing.cpp +++ b/examples/tutorial_api_cpp/14_synchronous_custom_preprocessing.cpp @@ -4,7 +4,9 @@ // In this function, the user can implement its own pre-processing, i.e., his function will be called after the image // has been read by OpenPose but before OpenPose processes the frames. -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #include // OpenPose dependencies #include @@ -22,13 +24,18 @@ public: void work(std::shared_ptr>>& datumsPtr) { - // User's pre-processing (after OpenPose read the input image & before OpenPose processing) here - // datumPtr->cvInputData: input frame try { + // User's pre-processing (after OpenPose read the input image & before OpenPose processing) here + // datumPtr->cvInputData: input frame if (datumsPtr != nullptr && !datumsPtr->empty()) + { for (auto& datumPtr : *datumsPtr) - cv::bitwise_not(datumPtr->cvOutputData, datumPtr->cvOutputData); + { + cv::Mat cvOutputData = OP_OP2CVMAT(datumPtr->cvOutputData); + cv::bitwise_not(cvOutputData, cvOutputData); + } + } } catch (const std::exception& e) { diff --git a/examples/tutorial_api_cpp/15_synchronous_custom_postprocessing.cpp b/examples/tutorial_api_cpp/15_synchronous_custom_postprocessing.cpp index b1cc182e..f698076b 100644 --- a/examples/tutorial_api_cpp/15_synchronous_custom_postprocessing.cpp +++ b/examples/tutorial_api_cpp/15_synchronous_custom_postprocessing.cpp @@ -4,7 +4,9 @@ // In this function, the user can implement its own post-processing, i.e., his function will be called after OpenPose // has processed the frames but before saving or visualizing any result. -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #include // OpenPose dependencies #include @@ -22,14 +24,19 @@ public: void work(std::shared_ptr>>& datumsPtr) { - // User's post-processing (after OpenPose processing & before OpenPose outputs) here - // datumPtr->cvOutputData: rendered frame with pose or heatmaps - // datumPtr->poseKeypoints: Array with the estimated pose try { + // User's post-processing (after OpenPose processing & before OpenPose outputs) here + // datumPtr->cvOutputData: rendered frame with pose or heatmaps + // datumPtr->poseKeypoints: Array with the estimated pose if (datumsPtr != nullptr && !datumsPtr->empty()) + { for (auto& datumPtr : *datumsPtr) - cv::bitwise_not(datumPtr->cvOutputData, datumPtr->cvOutputData); + { + cv::Mat cvOutputData = OP_OP2CVMAT(datumPtr->cvOutputData); + cv::bitwise_not(cvOutputData, cvOutputData); + } + } } catch (const std::exception& e) { diff --git a/examples/tutorial_api_cpp/16_synchronous_custom_output.cpp b/examples/tutorial_api_cpp/16_synchronous_custom_output.cpp index 65881d99..a4f2f50e 100644 --- a/examples/tutorial_api_cpp/16_synchronous_custom_output.cpp +++ b/examples/tutorial_api_cpp/16_synchronous_custom_output.cpp @@ -3,7 +3,9 @@ // performance. // In this function, the user can implement its own way to render/display/storage the results. -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include // OpenPose dependencies @@ -79,7 +81,8 @@ public: if (!FLAGS_no_display) { // Display rendered output image - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); // Display image and sleeps at least 1 ms (it usually sleeps ~5-10 msec to display the image) const char key = (char)cv::waitKey(1); if (key == 27) diff --git a/examples/tutorial_api_cpp/17_synchronous_custom_all_and_datum.cpp b/examples/tutorial_api_cpp/17_synchronous_custom_all_and_datum.cpp index 0e25f950..0086efe6 100644 --- a/examples/tutorial_api_cpp/17_synchronous_custom_all_and_datum.cpp +++ b/examples/tutorial_api_cpp/17_synchronous_custom_all_and_datum.cpp @@ -5,7 +5,9 @@ // function will be called after OpenPose has processed the frames but before saving), visualizing any result // render/display/storage the results, and use their custom Datum structure -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #define OPENPOSE_FLAGS_DISABLE_PRODUCER #define OPENPOSE_FLAGS_DISABLE_DISPLAY #include @@ -72,7 +74,8 @@ public: datumPtr = std::make_shared(); // Fill datum - datumPtr->cvInputData = cv::imread(mImageFiles.at(mCounter++)); + const cv::Mat cvInputData = cv::imread(mImageFiles.at(mCounter++)); + datumPtr->cvInputData = OP_CV2OPCONSTMAT(cvInputData); // If empty frame -> return nullptr if (datumPtr->cvInputData.empty()) @@ -112,14 +115,19 @@ public: void work(std::shared_ptr>>& datumsPtr) { - // User's post-processing (after OpenPose processing & before OpenPose outputs) here - // datumPtr->cvOutputData: rendered frame with pose or heatmaps - // datumPtr->poseKeypoints: Array with the estimated pose try { + // User's post-processing (after OpenPose processing & before OpenPose outputs) here + // datumPtr->cvOutputData: rendered frame with pose or heatmaps + // datumPtr->poseKeypoints: Array with the estimated pose if (datumsPtr != nullptr && !datumsPtr->empty()) + { for (auto& datumPtr : *datumsPtr) - cv::bitwise_not(datumPtr->cvOutputData, datumPtr->cvOutputData); + { + cv::Mat cvOutputData = OP_OP2CVMAT(datumPtr->cvOutputData); + cv::bitwise_not(cvOutputData, cvOutputData); + } + } } catch (const std::exception& e) { @@ -194,7 +202,8 @@ public: if (!FLAGS_no_display) { // Display rendered output image - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); // Display image and sleeps at least 1 ms (it usually sleeps ~5-10 msec to display the image) const char key = (char)cv::waitKey(1); if (key == 27) diff --git a/examples/tutorial_api_python/01_body_from_image.py b/examples/tutorial_api_python/01_body_from_image.py index 544d523f..7ca220e5 100644 --- a/examples/tutorial_api_python/01_body_from_image.py +++ b/examples/tutorial_api_python/01_body_from_image.py @@ -11,12 +11,12 @@ dir_path = os.path.dirname(os.path.realpath(__file__)) try: # Windows Import if platform == "win32": - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append(dir_path + '/../../python/openpose/Release'); os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' import pyopenpose as op else: - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append('../../python'); # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. # sys.path.append('/usr/local/python') diff --git a/examples/tutorial_api_python/02_whole_body_from_image.py b/examples/tutorial_api_python/02_whole_body_from_image.py index c573c961..ecec10d5 100644 --- a/examples/tutorial_api_python/02_whole_body_from_image.py +++ b/examples/tutorial_api_python/02_whole_body_from_image.py @@ -11,12 +11,12 @@ dir_path = os.path.dirname(os.path.realpath(__file__)) try: # Windows Import if platform == "win32": - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append(dir_path + '/../../python/openpose/Release'); os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' import pyopenpose as op else: - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append('../../python'); # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. # sys.path.append('/usr/local/python') diff --git a/examples/tutorial_api_python/08_heatmaps_from_image.py b/examples/tutorial_api_python/08_heatmaps_from_image.py index ee045fa8..18746f86 100644 --- a/examples/tutorial_api_python/08_heatmaps_from_image.py +++ b/examples/tutorial_api_python/08_heatmaps_from_image.py @@ -11,12 +11,12 @@ dir_path = os.path.dirname(os.path.realpath(__file__)) try: # Windows Import if platform == "win32": - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append(dir_path + '/../../python/openpose/Release'); os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' import pyopenpose as op else: - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append('../../python'); # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. # sys.path.append('/usr/local/python') diff --git a/examples/tutorial_api_python/09_keypoints_from_heatmaps.py b/examples/tutorial_api_python/09_keypoints_from_heatmaps.py index a406eaed..d82f2498 100644 --- a/examples/tutorial_api_python/09_keypoints_from_heatmaps.py +++ b/examples/tutorial_api_python/09_keypoints_from_heatmaps.py @@ -12,12 +12,12 @@ dir_path = os.path.dirname(os.path.realpath(__file__)) try: # Windows Import if platform == "win32": - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append(dir_path + '/../../python/openpose/Release'); os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' import pyopenpose as op else: - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append('../../python'); # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. # sys.path.append('/usr/local/python') diff --git a/examples/tutorial_api_python/openpose_python.py b/examples/tutorial_api_python/openpose_python.py index 9d2f0a3b..55c4333d 100644 --- a/examples/tutorial_api_python/openpose_python.py +++ b/examples/tutorial_api_python/openpose_python.py @@ -11,12 +11,12 @@ dir_path = os.path.dirname(os.path.realpath(__file__)) try: # Windows Import if platform == "win32": - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append(dir_path + '/../../python/openpose/Release'); os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' import pyopenpose as op else: - # Change these variables to point to the correct folder (Release/x64 etc.) + # Change these variables to point to the correct folder (Release/x64 etc.) sys.path.append('../../python'); # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. # sys.path.append('/usr/local/python') diff --git a/examples/tutorial_api_thread/1_thread_user_processing_function.cpp b/examples/tutorial_api_thread/1_thread_user_processing_function.cpp index 6a13475a..61e8e50f 100644 --- a/examples/tutorial_api_thread/1_thread_user_processing_function.cpp +++ b/examples/tutorial_api_thread/1_thread_user_processing_function.cpp @@ -8,7 +8,9 @@ // 1. `core` module: for the Datum struct that the `thread` module sends between the queues // 2. `utilities` module: for the error & logging functions, i.e., op::error & op::log respectively -// Command-line user intraface +// Third-party dependencies +#include +// Command-line user interface #include // OpenPose dependencies #include @@ -33,9 +35,14 @@ public: // User's processing here // datumPtr->cvInputData: initial cv::Mat obtained from the frames producer (video, webcam, etc.) // datumPtr->cvOutputData: final cv::Mat to be displayed - if (datumsPtr != nullptr) + if (datumsPtr != nullptr && !datumsPtr->empty()) + { for (auto& datumPtr : *datumsPtr) - cv::bitwise_not(datumPtr->cvInputData, datumPtr->cvOutputData); + { + cv::Mat cvOutputData = OP_OP2CVMAT(datumPtr->cvOutputData); + cv::bitwise_not(cvOutputData, cvOutputData); + } + } } catch (const std::exception& e) { @@ -82,8 +89,8 @@ int openPoseTutorialThread1() videoSeekSharedPtr->first = false; videoSeekSharedPtr->second = 0; const op::Point producerSize{ - (int)producerSharedPtr->get(CV_CAP_PROP_FRAME_WIDTH), - (int)producerSharedPtr->get(CV_CAP_PROP_FRAME_HEIGHT)}; + (int)producerSharedPtr->get(op::getCvCapPropFrameWidth()), + (int)producerSharedPtr->get(op::getCvCapPropFrameHeight())}; // Step 4 - Setting thread workers && manager typedef std::shared_ptr>> TypedefDatumsSP; op::ThreadManager threadManager; diff --git a/examples/tutorial_api_thread/2_thread_user_input_processing_output_and_datum.cpp b/examples/tutorial_api_thread/2_thread_user_input_processing_output_and_datum.cpp index 0e76fad2..aac0e7fb 100644 --- a/examples/tutorial_api_thread/2_thread_user_input_processing_output_and_datum.cpp +++ b/examples/tutorial_api_thread/2_thread_user_input_processing_output_and_datum.cpp @@ -8,7 +8,8 @@ // 1. `core` module: for the Datum struct that the `thread` module sends between the queues // 2. `utilities` module: for the error & logging functions, i.e., op::error & op::log respectively -// 3rdparty dependencies +// Third-party dependencies +#include // GFlags: DEFINE_bool, _int32, _int64, _uint64, _double, _string #include // Allow Google Flags in Ubuntu 14 @@ -86,7 +87,8 @@ public: datumPtr = std::make_shared(); // Fill datum - datumPtr->cvInputData = cv::imread(mImageFiles.at(mCounter++)); + const cv::Mat cvInputData = cv::imread(mImageFiles.at(mCounter++)); + datumPtr->cvInputData = OP_CV2OPCONSTMAT(cvInputData); // If empty frame -> return nullptr if (datumPtr->cvInputData.empty()) @@ -127,14 +129,19 @@ public: void work(std::shared_ptr>>& datumsPtr) { - // User's post-processing (after OpenPose processing & before OpenPose outputs) here - // datumPtr->cvOutputData: rendered frame with pose or heatmaps - // datumPtr->poseKeypoints: Array with the estimated pose try { + // User's post-processing (after OpenPose processing & before OpenPose outputs) here + // datumPtr->cvOutputData: rendered frame with pose or heatmaps + // datumPtr->poseKeypoints: Array with the estimated pose if (datumsPtr != nullptr && !datumsPtr->empty()) + { for (auto& datumPtr : *datumsPtr) - cv::bitwise_not(datumPtr->cvInputData, datumPtr->cvOutputData); + { + cv::Mat cvOutputData = OP_OP2CVMAT(datumPtr->cvOutputData); + cv::bitwise_not(cvOutputData, cvOutputData); + } + } } catch (const std::exception& e) { @@ -160,7 +167,8 @@ public: // datumPtr->poseKeypoints: Array with the estimated pose if (datumsPtr != nullptr && !datumsPtr->empty()) { - cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial Thread API", datumsPtr->at(0)->cvOutputData); + const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); + cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial Thread API", cvMat); // It displays the image and sleeps at least 1 ms (it usually sleeps ~5-10 msec to display the image) cv::waitKey(1); } diff --git a/examples/user_code/CMakeLists.txt b/examples/user_code/CMakeLists.txt index 921da368..08a96959 100644 --- a/examples/user_code/CMakeLists.txt +++ b/examples/user_code/CMakeLists.txt @@ -1,5 +1,5 @@ # Uncomment these lines with your custom file names -# set(USER_CODE_FILES +# set(USER_CODE_FILES # ADD_HERE_YOUR_FILE1.cpp # ADD_HERE_YOUR_FILE1.hpp # ADD_HERE_YOUR_FILE2.cpp diff --git a/examples/user_code/README.md b/examples/user_code/README.md index d907d184..74fcc7b4 100644 --- a/examples/user_code/README.md +++ b/examples/user_code/README.md @@ -18,7 +18,7 @@ You can quickly add your custom code into this folder so that quick prototypes c cd build/ make -j`nproc` # Windows -# Close Visual Studio, re-run CMake, and re-compile the project in Visual Studio +# Close Visual Studio, re-run CMake, and re-compile the project in Visual Studio ``` 5. **Run step 4 every time that you make changes into your code**. diff --git a/include/openpose/3d/cameraParameterReader.hpp b/include/openpose/3d/cameraParameterReader.hpp index 9e3e7272..32d51db8 100644 --- a/include/openpose/3d/cameraParameterReader.hpp +++ b/include/openpose/3d/cameraParameterReader.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_3D_CAMERA_PARAMETER_READER_HPP #define OPENPOSE_3D_CAMERA_PARAMETER_READER_HPP -#include #include namespace op @@ -15,10 +14,10 @@ namespace op // cameraExtrinsics is optional explicit CameraParameterReader(const std::string& serialNumber, - const cv::Mat& cameraIntrinsics, - const cv::Mat& cameraDistortion, - const cv::Mat& cameraExtrinsics = cv::Mat(), - const cv::Mat& cameraExtrinsicsInitial = cv::Mat()); + const Matrix& cameraIntrinsics, + const Matrix& cameraDistortion, + const Matrix& cameraExtrinsics = Matrix(), + const Matrix& cameraExtrinsicsInitial = Matrix()); // serialNumbers is optional. If empty, it will load all the XML files available in the // cameraParameterPath folder @@ -35,34 +34,27 @@ namespace op const std::vector& getCameraSerialNumbers() const; - const std::vector& getCameraMatrices() const; + const std::vector& getCameraMatrices() const; - const std::vector& getCameraDistortions() const; + const std::vector& getCameraDistortions() const; - const std::vector& getCameraIntrinsics() const; + const std::vector& getCameraIntrinsics() const; - const std::vector& getCameraExtrinsics() const; + const std::vector& getCameraExtrinsics() const; - const std::vector& getCameraExtrinsicsInitial() const; + const std::vector& getCameraExtrinsicsInitial() const; bool getUndistortImage() const; void setUndistortImage(const bool undistortImage); - void undistort(cv::Mat& frame, const unsigned int cameraIndex = 0u); + void undistort(Matrix& frame, const unsigned int cameraIndex = 0u); private: - std::vector mSerialNumbers; - std::vector mCameraMatrices; - std::vector mCameraDistortions; - std::vector mCameraIntrinsics; - std::vector mCameraExtrinsics; - std::vector mCameraExtrinsicsInitial; - - // Undistortion (optional) - bool mUndistortImage; - std::vector mRemoveDistortionMaps1; - std::vector mRemoveDistortionMaps2; + // PIMPL idiom + // http://www.cppsamples.com/common-tasks/pimpl.html + struct ImplCameraParameterReader; + std::shared_ptr spImpl; DELETE_COPY(CameraParameterReader); }; diff --git a/include/openpose/3d/poseTriangulation.hpp b/include/openpose/3d/poseTriangulation.hpp index f0f8e703..aa50dce0 100644 --- a/include/openpose/3d/poseTriangulation.hpp +++ b/include/openpose/3d/poseTriangulation.hpp @@ -1,29 +1,10 @@ #ifndef OPENPOSE_3D_POSE_TRIANGULATION_HPP #define OPENPOSE_3D_POSE_TRIANGULATION_HPP -#include #include namespace op { - /** - * 3D triangulation given known camera parameter matrices and based on linear DLT algorithm. - * The returned cv::Mat is a 4x1 matrix, where the last coordinate is 1. - */ - OP_API double triangulate( - cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, - const std::vector& pointsOnEachCamera); - - /** - * 3D triangulation given known camera parameter matrices and based on linear DLT algorithm with additional LMA - * non-linear refinement. - * The returned cv::Mat is a 4x1 matrix, where the last coordinate is 1. - * Note: If Ceres is not enabled, the LMA refinement is skipped and this function is equivalent to triangulate(). - */ - OP_API double triangulateWithOptimization( - cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, - const std::vector& pointsOnEachCamera, const double reprojectionMaxAcceptable); - class OP_API PoseTriangulation { public: @@ -34,11 +15,11 @@ namespace op void initializationOnThread(); Array reconstructArray( - const std::vector>& keypointsVector, const std::vector& cameraMatrices, + const std::vector>& keypointsVector, const std::vector& cameraMatrices, const std::vector>& imageSizes) const; std::vector> reconstructArray( - const std::vector>>& keypointsVector, const std::vector& cameraMatrices, + const std::vector>>& keypointsVector, const std::vector& cameraMatrices, const std::vector>& imageSizes) const; private: diff --git a/include/openpose/3d/wPoseTriangulation.hpp b/include/openpose/3d/wPoseTriangulation.hpp index f49ac904..d72fd037 100644 --- a/include/openpose/3d/wPoseTriangulation.hpp +++ b/include/openpose/3d/wPoseTriangulation.hpp @@ -70,7 +70,7 @@ namespace op // Profiling speed const auto profilerKey = Profiler::timerInit(__LINE__, __FUNCTION__, __FILE__); // 3-D triangulation and reconstruction - std::vector cameraMatrices; + std::vector cameraMatrices; std::vector> poseKeypointVector; std::vector> faceKeypointVector; std::vector> leftHandKeypointVector; @@ -84,7 +84,7 @@ namespace op rightHandKeypointVector.emplace_back(tDatumPtr->handKeypoints[1]); cameraMatrices.emplace_back(tDatumPtr->cameraMatrix); imageSizes.emplace_back( - Point{tDatumPtr->cvInputData.cols, tDatumPtr->cvInputData.rows}); + Point{tDatumPtr->cvInputData.cols(), tDatumPtr->cvInputData.rows()}); } // Pose 3-D reconstruction auto poseKeypoints3Ds = spPoseTriangulation->reconstructArray( diff --git a/include/openpose/calibration/headers.hpp b/include/openpose/calibration/headers.hpp index f6811d85..55d0cc50 100644 --- a/include/openpose/calibration/headers.hpp +++ b/include/openpose/calibration/headers.hpp @@ -3,6 +3,5 @@ // calibration module #include -#include #endif // OPENPOSE_CALIBRATION_HEADERS_HPP diff --git a/include/openpose/core/array.hpp b/include/openpose/core/array.hpp index 1ad7ff5e..c3a1413b 100644 --- a/include/openpose/core/array.hpp +++ b/include/openpose/core/array.hpp @@ -3,8 +3,8 @@ #include // std::shared_ptr #include -#include // cv::Mat #include +#include #include namespace op @@ -12,8 +12,8 @@ namespace op /** * Array: The OpenPose Basic Raw Data Container * This template class implements a multidimensional data array. It is our basic data container, analogous to - * cv::Mat in OpenCV, Tensor in Torch/TensorFlow or Blob in Caffe. - * It wraps a cv::Mat and a std::shared_ptr, both of them pointing to the same raw data. I.e. they both share the + * Mat in OpenCV, Tensor in Torch/TensorFlow or Blob in Caffe. + * It wraps a Matrix and a std::shared_ptr, both of them pointing to the same raw data. I.e. they both share the * same memory, so we can read and modify this data in both formats with no performance impact. * Hence, it keeps high performance while adding high-level functions. */ @@ -207,9 +207,9 @@ namespace op /** * Data allocation function. * It internally allocates memory and copies the data of the argument to the Array allocated memory. - * @param cvMat cv::Mat to be copied. + * @param cvMat Matrix to be copied. */ - void setFrom(const cv::Mat& cvMat); + void setFrom(const Matrix& cvMat); /** * Data allocation function. @@ -332,27 +332,27 @@ namespace op } /** - * Return a cv::Mat wrapper to the data. It forbids the data to be modified. + * Return a Matrix wrapper to the data. It forbids the data to be modified. * OpenCV only admits unsigned char, signed char, int, float & double. If the T class is not supported by * OpenCV, it will throw an error. - * Note: Array does not return an editable cv::Mat because some OpenCV functions reallocate memory and it + * Note: Array does not return an editable Matrix because some OpenCV functions reallocate memory and it * would not longer point to the Array instance. - * If you want to perform some OpenCV operation on the Array data, you can use: + * If you want to perform some OpenCV operation on the Array data, you can use: * editedCvMat = array.getConstCvMat().clone(); * // modify data * array.setFrom(editedCvMat) - * @return A const cv::Mat pointing to the data. + * @return A const Matrix pointing to the data. */ - const cv::Mat& getConstCvMat() const; + const Matrix& getConstCvMat() const; /** - * Analogous to getConstCvMat, but in this case it returns a editable cv::Mat. + * Analogous to getConstCvMat, but in this case it returns a editable Matrix. * Very important: Only allowed functions which do not provoke data reallocation. - * E.g., resizing functions will not work and they would provoke an undefined behaviour and/or execution + * E.g., resizing functions will not work and they would provoke an undefined behavior and/or execution * crashes. - * @return A cv::Mat pointing to the data. + * @return A Matrix pointing to the data. */ - cv::Mat& getCvMat(); + Matrix& getCvMat(); /** * [] operator @@ -479,7 +479,7 @@ namespace op size_t mVolume; std::shared_ptr spData; T* pData; // pData is a wrapper of spData. Used for Pybind11 binding. - std::pair mCvMatData; + std::pair mCvMatData; /** * Auxiliar function that both operator[](const std::vector& indexes) and diff --git a/include/openpose/core/common.hpp b/include/openpose/core/common.hpp index da05f9ed..19a2dfed 100644 --- a/include/openpose/core/common.hpp +++ b/include/openpose/core/common.hpp @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include diff --git a/include/openpose/core/cvMatToOpInput.hpp b/include/openpose/core/cvMatToOpInput.hpp index f5cbfc25..7480051e 100644 --- a/include/openpose/core/cvMatToOpInput.hpp +++ b/include/openpose/core/cvMatToOpInput.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_CORE_CV_MAT_TO_OP_INPUT_HPP #define OPENPOSE_CORE_CV_MAT_TO_OP_INPUT_HPP -#include // cv::Mat #include #include @@ -15,7 +14,7 @@ namespace op virtual ~CvMatToOpInput(); std::vector> createArray( - const cv::Mat& cvInputData, const std::vector& scaleInputToNetInputs, + const Matrix& inputData, const std::vector& scaleInputToNetInputs, const std::vector>& netInputSizes); private: diff --git a/include/openpose/core/cvMatToOpOutput.hpp b/include/openpose/core/cvMatToOpOutput.hpp index a0d913b7..0644d74a 100644 --- a/include/openpose/core/cvMatToOpOutput.hpp +++ b/include/openpose/core/cvMatToOpOutput.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_CORE_CV_MAT_TO_OP_OUTPUT_HPP #define OPENPOSE_CORE_CV_MAT_TO_OP_OUTPUT_HPP -#include // cv::Mat #include namespace op @@ -17,7 +16,7 @@ namespace op getSharedParameters(); Array createArray( - const cv::Mat& cvInputData, const double scaleInputToOutput, const Point& outputResolution); + const Matrix& inputData, const double scaleInputToOutput, const Point& outputResolution); private: const bool mGpuResize; diff --git a/include/openpose/core/datum.hpp b/include/openpose/core/datum.hpp index 438496a7..fe2c6573 100644 --- a/include/openpose/core/datum.hpp +++ b/include/openpose/core/datum.hpp @@ -1,10 +1,11 @@ #ifndef OPENPOSE_CORE_DATUM_HPP #define OPENPOSE_CORE_DATUM_HPP -#ifdef USE_EIGEN - #include +#ifdef USE_3D_ADAM_MODEL + #ifdef USE_EIGEN + #include + #endif #endif -#include // cv::Mat #include namespace op @@ -41,7 +42,7 @@ namespace op * Original image to be processed in cv::Mat uchar format. * Size: (input_width x input_height) x 3 channels */ - cv::Mat cvInputData; + Matrix cvInputData; /** * Original image to be processed in Array format. @@ -68,12 +69,12 @@ namespace op * If outputData is empty, cvOutputData will also be empty. * Size: (output_height x output_width) x 3 channels */ - cv::Mat cvOutputData; + Matrix cvOutputData; /** * Rendered 3D image in cv::Mat uchar format. */ - cv::Mat cvOutputData3D; + Matrix cvOutputData3D; // ------------------------------ Resulting Array data parameters ------------------------------ // /** @@ -195,17 +196,17 @@ namespace op /** * 3x4 camera matrix of the camera (equivalent to cameraIntrinsics * cameraExtrinsics). */ - cv::Mat cameraMatrix; + Matrix cameraMatrix; /** * 3x4 extrinsic parameters of the camera. */ - cv::Mat cameraExtrinsics; + Matrix cameraExtrinsics; /** * 3x3 intrinsic parameters of the camera. */ - cv::Mat cameraIntrinsics; + Matrix cameraIntrinsics; /** * If it is not empty, OpenPose will not run its internal body pose estimation network and will instead use @@ -223,7 +224,7 @@ namespace op /** * Size(s) (width x height) of the image(s) fed to the pose deep net. - * The size of the std::vector corresponds to the number of scales. + * The size of the std::vector corresponds to the number of scales. */ std::vector> netInputSizes; diff --git a/include/openpose/core/headers.hpp b/include/openpose/core/headers.hpp index c85f4512..8be24010 100644 --- a/include/openpose/core/headers.hpp +++ b/include/openpose/core/headers.hpp @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include diff --git a/include/openpose/core/macros.hpp b/include/openpose/core/macros.hpp index a32c5e49..0938acb7 100644 --- a/include/openpose/core/macros.hpp +++ b/include/openpose/core/macros.hpp @@ -7,7 +7,6 @@ #include #include // std::this_thread #include -#include // cv::Mat, check OpenCV version // OpenPose name and version const std::string OPEN_POSE_NAME_STRING = "OpenPose"; @@ -25,8 +24,7 @@ const std::string OPEN_POSE_NAME_AND_VERSION = OPEN_POSE_NAME_STRING + " " + OPE // Disable some Windows Warnings #ifdef _WIN32 - #pragma warning ( disable : 4251 ) // XXX needs to have dll-interface to be used by clients of class YYY - #pragma warning( disable: 4275 ) // non dll-interface structXXX used as base + #pragma warning(disable: 4251) // 'XXX': class 'YYY' needs to have dll-interface to be used by clients of class 'ZZZ' #endif #define UNUSED(unusedVariable) (void)(unusedVariable) @@ -87,59 +85,4 @@ const std::string OPEN_POSE_NAME_AND_VERSION = OPEN_POSE_NAME_STRING + " " + OPE // stackoverflow.com/questions/13978775/how-to-avoid-include-dependency-to-external-library?answertab=active#tab-top struct dim3; -// Compabitility for OpenCV 4.0 while preserving 2.4.X and 3.X compatibility -// Note: -// - CV_VERSION: 2.4.9.1 | 4.0.0-beta -// - CV_MAJOR_VERSION: 2 | 4 -// - CV_MINOR_VERSION: 4 | 0 -// - CV_SUBMINOR_VERSION: 9 | 0 -// - CV_VERSION_EPOCH: 2 | Not defined -#if (defined(CV_MAJOR_VERSION) && CV_MAJOR_VERSION > 3) - #define OPEN_CV_IS_4_OR_HIGHER -#endif -#ifdef OPEN_CV_IS_4_OR_HIGHER - #define CV_BGR2GRAY cv::COLOR_BGR2GRAY - #define CV_BGR2RGB cv::COLOR_BGR2RGB - #define CV_CALIB_CB_ADAPTIVE_THRESH cv::CALIB_CB_ADAPTIVE_THRESH - #define CV_CALIB_CB_NORMALIZE_IMAGE cv::CALIB_CB_NORMALIZE_IMAGE - #define CV_CALIB_CB_FILTER_QUADS cv::CALIB_CB_FILTER_QUADS - #define CV_CAP_PROP_FPS cv::CAP_PROP_FPS - #define CV_CAP_PROP_FRAME_COUNT cv::CAP_PROP_FRAME_COUNT - #define CV_CAP_PROP_FRAME_HEIGHT cv::CAP_PROP_FRAME_HEIGHT - #define CV_CAP_PROP_FRAME_WIDTH cv::CAP_PROP_FRAME_WIDTH - #define CV_CAP_PROP_POS_FRAMES cv::CAP_PROP_POS_FRAMES - #define CV_FOURCC cv::VideoWriter::fourcc - #define CV_GRAY2BGR cv::COLOR_GRAY2BGR - #define CV_HAAR_SCALE_IMAGE cv::CASCADE_SCALE_IMAGE - #define CV_INTER_CUBIC cv::INTER_CUBIC - #define CV_INTER_LINEAR cv::INTER_LINEAR - #define CV_L2 cv::NORM_L2 - #define CV_RGB2BGR cv::COLOR_RGB2BGR - #define CV_TERMCRIT_EPS cv::TermCriteria::Type::EPS - #define CV_TERMCRIT_ITER cv::TermCriteria::Type::MAX_ITER - #define CV_WARP_INVERSE_MAP cv::WARP_INVERSE_MAP - #define CV_WINDOW_FULLSCREEN cv::WINDOW_FULLSCREEN - #define CV_WINDOW_KEEPRATIO cv::WINDOW_KEEPRATIO - #define CV_WINDOW_NORMAL cv::WINDOW_NORMAL - #define CV_WINDOW_OPENGL cv::WINDOW_OPENGL - #define CV_WND_PROP_FULLSCREEN cv::WND_PROP_FULLSCREEN - // Required for alpha and beta versions, but not for rc version - #include - #ifndef CV_IMWRITE_JPEG_QUALITY - #define CV_IMWRITE_JPEG_QUALITY cv::IMWRITE_JPEG_QUALITY - #endif - #ifndef CV_IMWRITE_PNG_COMPRESSION - #define CV_IMWRITE_PNG_COMPRESSION cv::IMWRITE_PNG_COMPRESSION - #endif - #ifndef CV_LOAD_IMAGE_ANYDEPTH - #define CV_LOAD_IMAGE_ANYDEPTH cv::IMREAD_ANYDEPTH - #endif - #ifndef CV_LOAD_IMAGE_COLOR - #define CV_LOAD_IMAGE_COLOR cv::IMREAD_COLOR - #endif - #ifndef CV_LOAD_IMAGE_GRAYSCALE - #define CV_LOAD_IMAGE_GRAYSCALE cv::IMREAD_GRAYSCALE - #endif -#endif - #endif // OPENPOSE_CORE_MACROS_HPP diff --git a/include/openpose/core/matrix.hpp b/include/openpose/core/matrix.hpp new file mode 100644 index 00000000..0a4f3724 --- /dev/null +++ b/include/openpose/core/matrix.hpp @@ -0,0 +1,187 @@ +#ifndef OPENPOSE_CORE_MAT_HPP +#define OPENPOSE_CORE_MAT_HPP + +#include // std::shared_ptr +#include + +namespace op +{ + // Convert from Mat into cv::Mat. Usage example: + // #include + // ... + // cv::Mat opMat = OP2CVMAT(cv::Mat()); + #define OP_OP2CVMAT(opMat) \ + (*((cv::Mat*)((opMat).getCvMat()))) + + // Convert from Mat into const cv::Mat. Usage example: + // #include + // ... + // cv::Mat opMat = OP2CVCONSTMAT(cv::Mat()); + #define OP_OP2CVCONSTMAT(opMat) \ + (*((cv::Mat*)((opMat).getConstCvMat()))) + + // Convert from cv::Mat into Mat. Usage example: + // #include + // ... + // Mat opMat = CV2OPMAT(Mat()); + #define OP_CV2OPMAT(cvMat) \ + (op::Matrix((void*)&(cvMat))) + + // Convert from cv::Mat into const Mat. Usage example: + // #include + // ... + // Mat opMat = CV2OPCONSTMAT(Mat()); + #define OP_CV2OPCONSTMAT(cvMat) \ + (op::Matrix((const void* const)&(cvMat))) + + // Convert from std::vector into std::vector. Usage example: + // #include + // ... + // std::vector opMats; // Assume filled + // OP_OP2CVVECTORMAT(cvMats, opMats); + #define OP_OP2CVVECTORMAT(cvMats, opMats) \ + std::vector cvMats; \ + for (auto& opMat : (opMats)) \ + { \ + const auto cvMat = OP_OP2CVCONSTMAT(opMat); \ + cvMats.emplace_back(cvMat); \ + } + + // Convert from std::vector into std::vector. Usage example: + // #include + // ... + // std::vector cvMats; // Assume filled + // OP_CV2OPVECTORMAT(opMats, cvMats); + #define OP_CV2OPVECTORMAT(opMats, cvMats) \ + std::vector opMats; \ + for (auto& cvMat : (cvMats)) \ + { \ + const auto opMat = OP_CV2OPMAT(cvMat); \ + opMats.emplace_back(opMat); \ + } + + // Convert from std::vector into std::vector. Usage example: + // #include + // ... + // // Equivalents: + // OP_CV_VOID_FUNCTION(opMat, size()); + // // and + // OP_OP2CVMAT(cvMat, opMat); + // cvMat.size(); + #define OP_MAT_VOID_FUNCTION(opMat, function) \ + { \ + cv::Mat cvMat = OP_OP2CVMAT(cvMat, opMat); \ + cvMat.function; \ + } + #define OP_CONST_MAT_VOID_FUNCTION(opMat, function) \ + { \ + const cv::Mat cvMat = OP_OP2CVCONSTMAT(opMat); \ + cvMat.function; \ + } + #define OP_MAT_RETURN_FUNCTION(outputVariable, opMat, function) \ + { \ + cv::Mat cvMat = OP_OP2CVMAT(cvMat, opMat); \ + outputVariable = cvMat.function; \ + } + #define OP_CONST_MAT_RETURN_FUNCTION(outputVariable, opMat, function) \ + { \ + const cv::Mat cvMat = OP_OP2CVCONSTMAT(opMat); \ + outputVariable = cvMat.function; \ + } + + /** + * Mat: Bind of cv::Mat to avoid OpenCV as dependency in the headers. + */ + class OP_API Matrix + { + public: + Matrix(); + + /** + * @param cvMatPtr should be a cv::Mat element or it will provoke a core dumped. Done to + * avoid explicitly exposing 3rdparty libraries on the headers. + */ + explicit Matrix(const void* cvMatPtr); + + /** + * Analog to cv::Mat(int rows, int cols, int type, void *data, size_t step=AUTO_STEP) + * Very important: This Matrix will only "borrow" this pointer, so the caller must make sure to maintain the + * memory allocated until this Matrix destructor is called and also to handle the ucharPtr memory deallocation. + * @param ucharPtr should be a cv::Mat::data (or analog) element or it will provoke a core dumped. Done to + * avoid explicitly exposing 3rdparty libraries on the headers. + */ + explicit Matrix(const int rows, const int cols, const int type, void* cvMatPtr); + + Matrix clone() const; + + /** + * @return cv::Mat*. + */ + void* getCvMat(); + + /** + * @return const cv::Mat*. + */ + const void* getConstCvMat() const; + + /** + * Equivalent to cv::Mat::data + */ + + unsigned char* data(); + /** + * Equivalent to cv::Mat::data + */ + const unsigned char* dataConst() const; + + /** + * Equivalent to cv::Mat::eye + */ + static Matrix eye(const int rows, const int cols, const int type); + /** + * Equivalent to cv::Mat::cols + */ + int cols() const; + /** + * Equivalent to cv::Mat::rows + */ + int rows() const; + /** + * Equivalent to cv::Mat::size[dimension] + */ + int size(const int dimension) const; + /** + * Equivalent to cv::Mat::dims + */ + int dims() const; + + /** + * Equivalent to their analog cv::Mat functions + */ + bool isContinuous() const; + bool isSubmatrix() const; + size_t elemSize() const; + size_t elemSize1() const; + int type() const; + int depth() const; + int channels() const; + size_t step1(const int i = 0) const; + bool empty() const; + size_t total() const; + int checkVector(const int elemChannels, const int depth = -1, const bool requireContinuous = true) const; + + /** + * Similar to their analog cv::Mat functions + */ + void setTo(const double value); + void copyTo(Matrix& outputMat) const; + + private: + // PIMPL idiom + // http://www.cppsamples.com/common-tasks/pimpl.html + struct ImplMat; + std::shared_ptr spImpl; + }; +} + +#endif // OPENPOSE_CORE_MAT_HPP diff --git a/include/openpose/core/opOutputToCvMat.hpp b/include/openpose/core/opOutputToCvMat.hpp index f11b6aba..77eb2b4c 100644 --- a/include/openpose/core/opOutputToCvMat.hpp +++ b/include/openpose/core/opOutputToCvMat.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_CORE_OP_OUTPUT_TO_CV_MAT_HPP #define OPENPOSE_CORE_OP_OUTPUT_TO_CV_MAT_HPP -#include // cv::Mat #include namespace op @@ -16,7 +15,7 @@ namespace op void setSharedParameters( const std::tuple, std::shared_ptr, std::shared_ptr>& tuple); - cv::Mat formatToCvMat(const Array& outputData); + Matrix formatToCvMat(const Array& outputData); private: const bool mGpuResize; diff --git a/include/openpose/core/wKeypointScaler.hpp b/include/openpose/core/wKeypointScaler.hpp index 9930207d..ed4676c3 100644 --- a/include/openpose/core/wKeypointScaler.hpp +++ b/include/openpose/core/wKeypointScaler.hpp @@ -67,11 +67,11 @@ namespace op tDatumPtr->handKeypoints[1], tDatumPtr->faceKeypoints}; spKeypointScaler->scale( arraysToScale, tDatumPtr->scaleInputToOutput, tDatumPtr->scaleNetToOutput, - Point{tDatumPtr->cvInputData.cols, tDatumPtr->cvInputData.rows}); + Point{tDatumPtr->cvInputData.cols(), tDatumPtr->cvInputData.rows()}); // Rescale part candidates spKeypointScaler->scale( tDatumPtr->poseCandidates, tDatumPtr->scaleInputToOutput, tDatumPtr->scaleNetToOutput, - Point{tDatumPtr->cvInputData.cols, tDatumPtr->cvInputData.rows}); + Point{tDatumPtr->cvInputData.cols(), tDatumPtr->cvInputData.rows()}); } // Profiling speed Profiler::timerEnd(profilerKey); diff --git a/include/openpose/core/wScaleAndSizeExtractor.hpp b/include/openpose/core/wScaleAndSizeExtractor.hpp index 57c526ec..57929f01 100644 --- a/include/openpose/core/wScaleAndSizeExtractor.hpp +++ b/include/openpose/core/wScaleAndSizeExtractor.hpp @@ -65,7 +65,7 @@ namespace op // cv::Mat -> float* for (auto& tDatumPtr : *tDatums) { - const Point inputSize{tDatumPtr->cvInputData.cols, tDatumPtr->cvInputData.rows}; + const Point inputSize{tDatumPtr->cvInputData.cols(), tDatumPtr->cvInputData.rows()}; std::tie(tDatumPtr->scaleInputToNetInputs, tDatumPtr->netInputSizes, tDatumPtr->scaleInputToOutput, tDatumPtr->netOutputSize) = spScaleAndSizeExtractor->extract(inputSize); } diff --git a/include/openpose/face/faceDetectorOpenCV.hpp b/include/openpose/face/faceDetectorOpenCV.hpp index 38a6d521..577aebef 100644 --- a/include/openpose/face/faceDetectorOpenCV.hpp +++ b/include/openpose/face/faceDetectorOpenCV.hpp @@ -1,8 +1,6 @@ #ifndef OPENPOSE_FACE_FACE_DETECTOR_OPENCV_HPP #define OPENPOSE_FACE_FACE_DETECTOR_OPENCV_HPP -#include -#include #include namespace op @@ -15,10 +13,13 @@ namespace op virtual ~FaceDetectorOpenCV(); // No thread-save - std::vector> detectFaces(const cv::Mat& cvInputData); + std::vector> detectFaces(const Matrix& inputData); private: - cv::CascadeClassifier mFaceCascade; + // PIMPL idiom + // http://www.cppsamples.com/common-tasks/pimpl.html + struct ImplFaceDetectorOpenCV; + std::unique_ptr upImpl; DELETE_COPY(FaceDetectorOpenCV); }; diff --git a/include/openpose/face/faceExtractorCaffe.hpp b/include/openpose/face/faceExtractorCaffe.hpp index 96c8c085..513f8360 100644 --- a/include/openpose/face/faceExtractorCaffe.hpp +++ b/include/openpose/face/faceExtractorCaffe.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_FACE_FACE_EXTRACTOR_CAFFE_HPP #define OPENPOSE_FACE_FACE_EXTRACTOR_CAFFE_HPP -#include // cv::Mat #include #include #include @@ -39,9 +38,9 @@ namespace op * each index corresponds to a different person in the image. Internally, a op::Rectangle * (similar to cv::Rect for floating values) with the position of that face (or 0,0,0,0 if * some face is missing, e.g., if a specific person has only half of the body inside the image). - * @param cvInputData Original image in cv::Mat format and BGR format. + * @param cvInputData Original image in Mat format and BGR format. */ - void forwardPass(const std::vector>& faceRectangles, const cv::Mat& cvInputData); + void forwardPass(const std::vector>& faceRectangles, const Matrix& inputData); private: // PIMPL idiom diff --git a/include/openpose/face/faceExtractorNet.hpp b/include/openpose/face/faceExtractorNet.hpp index 2c691efc..e92a49ac 100644 --- a/include/openpose/face/faceExtractorNet.hpp +++ b/include/openpose/face/faceExtractorNet.hpp @@ -2,7 +2,6 @@ #define OPENPOSE_FACE_FACE_EXTRACTOR_HPP #include -#include // cv::Mat #include #include @@ -41,9 +40,9 @@ namespace op * each index corresponds to a different person in the image. Internally, a op::Rectangle * (similar to cv::Rect for floating values) with the position of that face (or 0,0,0,0 if * some face is missing, e.g., if a specific person has only half of the body inside the image). - * @param cvInputData Original image in cv::Mat format and BGR format. + * @param cvInputData Original image in Mat format and BGR format. */ - virtual void forwardPass(const std::vector>& faceRectangles, const cv::Mat& cvInputData) = 0; + virtual void forwardPass(const std::vector>& faceRectangles, const Matrix& inputData) = 0; Array getHeatMaps() const; diff --git a/include/openpose/filestream/fileStream.hpp b/include/openpose/filestream/fileStream.hpp index 3e84b085..8cd9e7b6 100644 --- a/include/openpose/filestream/fileStream.hpp +++ b/include/openpose/filestream/fileStream.hpp @@ -1,10 +1,9 @@ #ifndef OPENPOSE_FILESTREAM_FILE_STREAM_HPP #define OPENPOSE_FILESTREAM_FILE_STREAM_HPP -#include // cv::Mat -#include // CV_LOAD_IMAGE_ANYDEPTH, CV_IMWRITE_PNG_COMPRESSION #include #include +#include namespace op { @@ -26,18 +25,18 @@ namespace op // Save/load json, xml, yaml, yml OP_API void saveData( - const std::vector& cvMats, const std::vector& cvMatNames, + const std::vector& opMats, const std::vector& cvMatNames, const std::string& fileNameNoExtension, const DataFormat dataFormat); OP_API void saveData( - const cv::Mat& cvMat, const std::string cvMatName, const std::string& fileNameNoExtension, + const Matrix& opMat, const std::string cvMatName, const std::string& fileNameNoExtension, const DataFormat dataFormat); - OP_API std::vector loadData( + OP_API std::vector loadData( const std::vector& cvMatNames, const std::string& fileNameNoExtension, const DataFormat dataFormat); - OP_API cv::Mat loadData( + OP_API Matrix loadData( const std::string& cvMatName, const std::string& fileNameNoExtension, const DataFormat dataFormat); // Json - Saving as *.json not available in OpenCV verions < 3.0, this function is a quick fix @@ -53,11 +52,11 @@ namespace op // Save/load image OP_API void saveImage( - const cv::Mat& cvMat, const std::string& fullFilePath, + const Matrix& matrix, const std::string& fullFilePath, const std::vector& openCvCompressionParams - = {CV_IMWRITE_JPEG_QUALITY, 100, CV_IMWRITE_PNG_COMPRESSION, 9}); + = {getCvImwriteJpegQuality(), 100, getCvImwritePngCompression(), 9}); - OP_API cv::Mat loadImage(const std::string& fullFilePath, const int openCvFlags = CV_LOAD_IMAGE_ANYDEPTH); + OP_API Matrix loadImage(const std::string& fullFilePath, const int openCvFlags = getCvLoadImageAnydepth()); OP_API std::vector, 2>> loadHandDetectorTxt(const std::string& txtFilePath); } diff --git a/include/openpose/filestream/imageSaver.hpp b/include/openpose/filestream/imageSaver.hpp index 5e51b3dc..0b1d9a0f 100644 --- a/include/openpose/filestream/imageSaver.hpp +++ b/include/openpose/filestream/imageSaver.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_FILESTREAM_IMAGE_SAVER_HPP #define OPENPOSE_FILESTREAM_IMAGE_SAVER_HPP -#include // cv::Mat #include #include @@ -14,9 +13,9 @@ namespace op virtual ~ImageSaver(); - void saveImages(const cv::Mat& cvOutputData, const std::string& fileName) const; + void saveImages(const Matrix& cvOutputData, const std::string& fileName) const; - void saveImages(const std::vector& cvOutputDatas, const std::string& fileName) const; + void saveImages(const std::vector& matOutputDatas, const std::string& fileName) const; private: const std::string mImageFormat; diff --git a/include/openpose/filestream/videoSaver.hpp b/include/openpose/filestream/videoSaver.hpp index a4cb4416..f300ed03 100644 --- a/include/openpose/filestream/videoSaver.hpp +++ b/include/openpose/filestream/videoSaver.hpp @@ -16,9 +16,9 @@ namespace op bool isOpened(); - void write(const cv::Mat& cvMat); + void write(const Matrix& matToSave); - void write(const std::vector& cvMats); + void write(const std::vector& matsToSave); private: // PIMPL idiom diff --git a/include/openpose/filestream/wImageSaver.hpp b/include/openpose/filestream/wImageSaver.hpp index d663b847..9d96c918 100644 --- a/include/openpose/filestream/wImageSaver.hpp +++ b/include/openpose/filestream/wImageSaver.hpp @@ -64,12 +64,12 @@ namespace op // T* to T auto& tDatumsNoPtr = *tDatums; // Record image(s) on disk - std::vector cvOutputDatas(tDatumsNoPtr.size()); + std::vector opOutputDatas(tDatumsNoPtr.size()); for (auto i = 0u; i < tDatumsNoPtr.size(); i++) - cvOutputDatas[i] = tDatumsNoPtr[i]->cvOutputData; + opOutputDatas[i] = tDatumsNoPtr[i]->cvOutputData; const auto fileName = (!tDatumsNoPtr[0]->name.empty() ? tDatumsNoPtr[0]->name : std::to_string(tDatumsNoPtr[0]->id)); - spImageSaver->saveImages(cvOutputDatas, fileName); + spImageSaver->saveImages(opOutputDatas, fileName); // Profiling speed Profiler::timerEnd(profilerKey); Profiler::printAveragedTimeMsOnIterationX(profilerKey, __LINE__, __FUNCTION__, __FILE__); diff --git a/include/openpose/filestream/wVideoSaver.hpp b/include/openpose/filestream/wVideoSaver.hpp index fdc979c1..fe176324 100644 --- a/include/openpose/filestream/wVideoSaver.hpp +++ b/include/openpose/filestream/wVideoSaver.hpp @@ -64,10 +64,10 @@ namespace op // T* to T auto& tDatumsNoPtr = *tDatums; // Record video(s) - std::vector cvOutputDatas(tDatumsNoPtr.size()); - for (auto i = 0u ; i < cvOutputDatas.size() ; i++) - cvOutputDatas[i] = tDatumsNoPtr[i]->cvOutputData; - spVideoSaver->write(cvOutputDatas); + std::vector opOutputDatas(tDatumsNoPtr.size()); + for (auto i = 0u ; i < opOutputDatas.size() ; i++) + opOutputDatas[i] = tDatumsNoPtr[i]->cvOutputData; + spVideoSaver->write(opOutputDatas); // Profiling speed Profiler::timerEnd(profilerKey); Profiler::printAveragedTimeMsOnIterationX(profilerKey, __LINE__, __FUNCTION__, __FILE__); diff --git a/include/openpose/gui/frameDisplayer.hpp b/include/openpose/gui/frameDisplayer.hpp index ae4417b4..e5347bd1 100644 --- a/include/openpose/gui/frameDisplayer.hpp +++ b/include/openpose/gui/frameDisplayer.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_GUI_FRAMES_DISPLAY_HPP #define OPENPOSE_GUI_FRAMES_DISPLAY_HPP -#include // cv::Mat #include #include @@ -43,18 +42,18 @@ namespace op /** * This function displays an image on the display. - * @param frame cv::Mat image to display. + * @param frame Mat image to display. * @param waitKeyValue int value that specifies the argument parameter for cv::waitKey (see OpenCV * documentation for more information). Special cases: select -1 * not to use cv::waitKey or 0 for cv::waitKey(0). OpenCV doc: * http://docs.opencv.org/2.4/modules/highgui/doc/user_interface.html?highlight=waitkey */ - void displayFrame(const cv::Mat& frame, const int waitKeyValue = -1); + void displayFrame(const Matrix& frame, const int waitKeyValue = -1); /** * Analogous to the previous displayFrame, but first it horizontally concatenates all the frames */ - void displayFrame(const std::vector& frames, const int waitKeyValue = -1); + void displayFrame(const std::vector& frames, const int waitKeyValue = -1); private: const std::string mWindowName; diff --git a/include/openpose/gui/gui.hpp b/include/openpose/gui/gui.hpp index 23c4463f..2f3aa050 100644 --- a/include/openpose/gui/gui.hpp +++ b/include/openpose/gui/gui.hpp @@ -2,7 +2,6 @@ #define OPENPOSE_GUI_GUI_HPP #include -#include // cv::Mat #include #include #include @@ -28,9 +27,9 @@ namespace op virtual void initializationOnThread(); - void setImage(const cv::Mat& cvMatOutput); + void setImage(const Matrix& cvMatOutput); - void setImage(const std::vector& cvMatOutputs); + void setImage(const std::vector& cvMatOutputs); virtual void update(); diff --git a/include/openpose/gui/gui3D.hpp b/include/openpose/gui/gui3D.hpp index ed298d32..61a2c4b4 100644 --- a/include/openpose/gui/gui3D.hpp +++ b/include/openpose/gui/gui3D.hpp @@ -32,7 +32,7 @@ namespace op virtual void update(); - virtual cv::Mat readCvMat(); + virtual Matrix readCvMat(); private: const bool mCopyGlToCvMat; diff --git a/include/openpose/gui/guiInfoAdder.hpp b/include/openpose/gui/guiInfoAdder.hpp index cdbfaa4c..f03e80fa 100644 --- a/include/openpose/gui/guiInfoAdder.hpp +++ b/include/openpose/gui/guiInfoAdder.hpp @@ -2,7 +2,6 @@ #define OPENPOSE_GUI_ADD_GUI_INFO_HPP #include -#include // cv::Mat #include namespace op @@ -14,7 +13,7 @@ namespace op virtual ~GuiInfoAdder(); - void addInfo(cv::Mat& cvOutputData, const int numberPeople, const unsigned long long id, + void addInfo(Matrix& outputData, const int numberPeople, const unsigned long long id, const std::string& elementRenderedName, const unsigned long long frameNumber, const Array& poseIds = Array{}, const Array& poseKeypoints = Array{}); diff --git a/include/openpose/gui/wGui.hpp b/include/openpose/gui/wGui.hpp index 6a3fe677..feb9fe21 100644 --- a/include/openpose/gui/wGui.hpp +++ b/include/openpose/gui/wGui.hpp @@ -73,7 +73,7 @@ namespace op // Update cvMat if (!tDatums->empty()) { - std::vector cvOutputDatas; + std::vector cvOutputDatas; for (auto& tDatumPtr : *tDatums) cvOutputDatas.emplace_back(tDatumPtr->cvOutputData); spGui->setImage(cvOutputDatas); diff --git a/include/openpose/gui/wGui3D.hpp b/include/openpose/gui/wGui3D.hpp index e8ca5d10..83dd6335 100644 --- a/include/openpose/gui/wGui3D.hpp +++ b/include/openpose/gui/wGui3D.hpp @@ -75,7 +75,7 @@ namespace op if (!tDatums->empty()) { // Update cvMat - std::vector cvOutputDatas; + std::vector cvOutputDatas; for (auto& tDatumPtr : *tDatums) cvOutputDatas.emplace_back(tDatumPtr->cvOutputData); spGui3D->setImage(cvOutputDatas); diff --git a/include/openpose/gui/wGuiAdam.hpp b/include/openpose/gui/wGuiAdam.hpp index 966b4c51..ab60537f 100644 --- a/include/openpose/gui/wGuiAdam.hpp +++ b/include/openpose/gui/wGuiAdam.hpp @@ -75,7 +75,7 @@ namespace op if (!tDatums->empty()) { // Update cvMat - std::vector cvOutputDatas; + std::vector cvOutputDatas; for (auto& tDatum : *tDatums) cvOutputDatas.emplace_back(tDatumPtr->cvOutputData); spGuiAdam->setImage(cvOutputDatas); diff --git a/include/openpose/hand/handExtractorCaffe.hpp b/include/openpose/hand/handExtractorCaffe.hpp index bea4a322..2d57367f 100644 --- a/include/openpose/hand/handExtractorCaffe.hpp +++ b/include/openpose/hand/handExtractorCaffe.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_HAND_HAND_EXTRACTOR_CAFFE_HPP #define OPENPOSE_HAND_HAND_EXTRACTOR_CAFFE_HPP -#include // cv::Mat #include #include #include @@ -26,7 +25,7 @@ namespace op */ HandExtractorCaffe(const Point& netInputSize, const Point& netOutputSize, const std::string& modelFolder, const int gpuId, - const unsigned short numberScales = 1, const float rangeScales = 0.4f, + const int numberScales = 1, const float rangeScales = 0.4f, const std::vector& heatMapTypes = {}, const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOne, const bool enableGoogleLogging = true); @@ -50,9 +49,9 @@ namespace op * elements: index 0 and 1 for left and right hand respectively. Inside each array element, a * op::Rectangle (similar to cv::Rect for floating values) with the position of that hand (or 0,0,0,0 if * some hand is missing, e.g., if a specific person has only half of the body inside the image). - * @param cvInputData Original image in cv::Mat format and BGR format. + * @param inputData Original image in Mat format and BGR format. */ - void forwardPass(const std::vector, 2>> handRectangles, const cv::Mat& cvInputData); + void forwardPass(const std::vector, 2>> handRectangles, const Matrix& inputData); private: // PIMPL idiom @@ -60,9 +59,6 @@ namespace op struct ImplHandExtractorCaffe; std::unique_ptr upImpl; - void detectHandKeypoints(Array& handCurrent, const int person, - const cv::Mat& affineMatrix); - Array getHeatMapsFromLastPass() const; // PIMP requires DELETE_COPY & destructor, or extra code diff --git a/include/openpose/hand/handExtractorNet.hpp b/include/openpose/hand/handExtractorNet.hpp index 7f8ac850..4c3767d6 100644 --- a/include/openpose/hand/handExtractorNet.hpp +++ b/include/openpose/hand/handExtractorNet.hpp @@ -2,7 +2,6 @@ #define OPENPOSE_HAND_HAND_EXTRACTOR_HPP #include -#include // cv::Mat #include #include @@ -23,7 +22,7 @@ namespace op * @param rangeScales The range between the smaller and bigger scale. */ explicit HandExtractorNet(const Point& netInputSize, const Point& netOutputSize, - const unsigned short numberScales = 1, const float rangeScales = 0.4f, + const int numberScales = 1, const float rangeScales = 0.4f, const std::vector& heatMapTypes = {}, const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOne); @@ -46,10 +45,10 @@ namespace op * elements: index 0 and 1 for left and right hand respectively. Inside each array element, a * op::Rectangle (similar to cv::Rect for floating values) with the position of that hand (or 0,0,0,0 if * some hand is missing, e.g., if a specific person has only half of the body inside the image). - * @param cvInputData Original image in cv::Mat format and BGR format. + * @param cvInputData Original image in Mat format and BGR format. */ virtual void forwardPass(const std::vector, 2>> handRectangles, - const cv::Mat& cvInputData) = 0; + const Matrix& cvInputData) = 0; std::array, 2> getHeatMaps() const; @@ -67,7 +66,7 @@ namespace op void setEnabled(const bool enabled); protected: - const std::pair mMultiScaleNumberAndRange; + const std::pair mMultiScaleNumberAndRange; const Point mNetOutputSize; Array mHandImageCrop; std::array, 2> mHandKeypoints; diff --git a/include/openpose/pose/poseExtractor.hpp b/include/openpose/pose/poseExtractor.hpp index 31a7855d..a4692ed7 100644 --- a/include/openpose/pose/poseExtractor.hpp +++ b/include/openpose/pose/poseExtractor.hpp @@ -46,20 +46,20 @@ namespace op // PersonIdExtractor functions // Not thread-safe - Array extractIds(const Array& poseKeypoints, const cv::Mat& cvMatInput, + Array extractIds(const Array& poseKeypoints, const Matrix& cvMatInput, const unsigned long long imageIndex = 0ull); // Same than extractIds but thread-safe - Array extractIdsLockThread(const Array& poseKeypoints, const cv::Mat& cvMatInput, + Array extractIdsLockThread(const Array& poseKeypoints, const Matrix& cvMatInput, const unsigned long long imageIndex, const long long frameId); // PersonTracker functions void track(Array& poseKeypoints, Array& poseIds, - const cv::Mat& cvMatInput, const unsigned long long imageViewIndex = 0ull); + const Matrix& cvMatInput, const unsigned long long imageViewIndex = 0ull); void trackLockThread(Array& poseKeypoints, Array& poseIds, - const cv::Mat& cvMatInput, + const Matrix& cvMatInput, const unsigned long long imageViewIndex, const long long frameId); diff --git a/include/openpose/pose/renderPose.hpp b/include/openpose/pose/renderPose.hpp index 196b3617..ace60250 100644 --- a/include/openpose/pose/renderPose.hpp +++ b/include/openpose/pose/renderPose.hpp @@ -1,7 +1,6 @@ #ifndef OPENPOSE_POSE_RENDER_POSE_HPP #define OPENPOSE_POSE_RENDER_POSE_HPP -#include // cv::Mat #include #include #include diff --git a/include/openpose/pose/wPoseExtractor.hpp b/include/openpose/pose/wPoseExtractor.hpp index 7d621ea0..6da487e7 100644 --- a/include/openpose/pose/wPoseExtractor.hpp +++ b/include/openpose/pose/wPoseExtractor.hpp @@ -76,7 +76,7 @@ namespace op auto& tDatumPtr = (*tDatums)[i]; // OpenPose net forward pass spPoseExtractor->forwardPass( - tDatumPtr->inputNetData, Point{tDatumPtr->cvInputData.cols, tDatumPtr->cvInputData.rows}, + tDatumPtr->inputNetData, Point{tDatumPtr->cvInputData.cols(), tDatumPtr->cvInputData.rows()}, tDatumPtr->scaleInputToNetInputs, tDatumPtr->poseNetOutput, tDatumPtr->id); // OpenPose keypoint detector tDatumPtr->poseCandidates = spPoseExtractor->getCandidatesCopy(); diff --git a/include/openpose/pose/wPoseExtractorNet.hpp b/include/openpose/pose/wPoseExtractorNet.hpp index 2e3eeea6..14662fde 100644 --- a/include/openpose/pose/wPoseExtractorNet.hpp +++ b/include/openpose/pose/wPoseExtractorNet.hpp @@ -73,7 +73,7 @@ namespace op for (auto& tDatumPtr : *tDatums) { spPoseExtractorNet->forwardPass( - tDatumPtr->inputNetData, Point{tDatumPtr->cvInputData.cols, tDatumPtr->cvInputData.rows}, + tDatumPtr->inputNetData, Point{tDatumPtr->cvInputData.cols(), tDatumPtr->cvInputData.rows()}, tDatumPtr->scaleInputToNetInputs, tDatumPtr->poseNetOutput); tDatumPtr->poseCandidates = spPoseExtractorNet->getCandidatesCopy(); tDatumPtr->poseHeatMaps = spPoseExtractorNet->getHeatMapsCopy(); diff --git a/include/openpose/producer/datumProducer.hpp b/include/openpose/producer/datumProducer.hpp index b36b25cc..062c65f3 100644 --- a/include/openpose/producer/datumProducer.hpp +++ b/include/openpose/producer/datumProducer.hpp @@ -3,7 +3,6 @@ #include #include // std::numeric_limits -#include #include #include #include @@ -45,10 +44,26 @@ namespace op // Implementation -#include // cv::cvtColor #include +#include namespace op { + // Auxiliary functions for DatumProducer in order to 1) Reduce compiling time and 2) Remove OpenCV deps. + OP_API void datumProducerConstructor( + const std::shared_ptr& producerSharedPtr, const unsigned long long frameFirst, + const unsigned long long frameStep, const unsigned long long frameLast); + OP_API void datumProducerConstructorTooManyConsecutiveEmptyFrames( + unsigned int& numberConsecutiveEmptyFrames, const bool emptyFrame); + OP_API bool datumProducerConstructorRunningAndGetDatumIsDatumProducerRunning( + const std::shared_ptr& producerSharedPtr, const unsigned long long numberFramesToProcess, + const unsigned long long globalCounter); + OP_API void datumProducerConstructorRunningAndGetDatumApplyPlayerControls( + const std::shared_ptr& producerSharedPtr, + const std::shared_ptr, std::atomic>>& videoSeekSharedPtr); + OP_API unsigned long long datumProducerConstructorRunningAndGetNextFrameNumber( + const std::shared_ptr& producerSharedPtr); + OP_API void datumProducerConstructorRunningAndGetDatumFrameIntegrity(Matrix& matrix); + template DatumProducer::DatumProducer( const std::shared_ptr& producerSharedPtr, @@ -65,26 +80,7 @@ namespace op { try { - // Sanity check - if (frameLast < frameFirst) - error("The desired initial frame must be lower than the last one (flags `--frame_first` vs." - " `--frame_last`). Current: " + std::to_string(frameFirst) + " vs. " + std::to_string(frameLast) - + ".", __LINE__, __FUNCTION__, __FILE__); - if (frameLast != std::numeric_limits::max() - && frameLast > spProducer->get(CV_CAP_PROP_FRAME_COUNT)-1) - error("The desired last frame must be lower than the length of the video or the number of images." - " Current: " + std::to_string(frameLast) + " vs. " - + std::to_string(positiveIntRound(spProducer->get(CV_CAP_PROP_FRAME_COUNT))-1) + ".", - __LINE__, __FUNCTION__, __FILE__); - // Set frame first and step - if (spProducer->getType() != ProducerType::FlirCamera && spProducer->getType() != ProducerType::IPCamera - && spProducer->getType() != ProducerType::Webcam) - { - // Frame first - spProducer->set(CV_CAP_PROP_POS_FRAMES, (double)frameFirst); - // Frame step - spProducer->set(ProducerProperty::FrameStep, (double)frameStep); - } + datumProducerConstructor(producerSharedPtr, frameFirst, frameStep, frameLast); } catch (const std::exception& e) { @@ -102,78 +98,59 @@ namespace op { try { - auto datums = std::make_shared>>(); - // Check last desired frame has not been reached - if (mNumberFramesToProcess != std::numeric_limits::max() - && mGlobalCounter > mNumberFramesToProcess) - { - spProducer->release(); - } - // If producer released -> it sends an empty cv::Mat + a datumProducerRunning signal - const bool datumProducerRunning = spProducer->isOpened(); + // If producer released -> it sends an empty Matrix + a datumProducerRunning signal + const bool datumProducerRunning = datumProducerConstructorRunningAndGetDatumIsDatumProducerRunning( + spProducer, mNumberFramesToProcess, mGlobalCounter); // If device is open + auto datums = std::make_shared>>(); if (datumProducerRunning) { // Fast forward/backward - Seek to specific frame index desired - if (spVideoSeek != nullptr) - { - // Fake pause vs. normal mode - const auto increment = spVideoSeek->second - (spVideoSeek->first ? 1 : 0); - // Normal mode - if (increment != 0) - spProducer->set(CV_CAP_PROP_POS_FRAMES, spProducer->get(CV_CAP_PROP_POS_FRAMES) + increment); - // It must be always reset or bug in fake pause - spVideoSeek->second = 0; - } - auto nextFrameName = spProducer->getNextFrameName(); - const auto nextFrameNumber = (unsigned long long)spProducer->get(CV_CAP_PROP_POS_FRAMES); - const auto cvMats = spProducer->getFrames(); - const auto cameraMatrices = spProducer->getCameraMatrices(); - auto cameraExtrinsics = spProducer->getCameraExtrinsics(); - auto cameraIntrinsics = spProducer->getCameraIntrinsics(); + datumProducerConstructorRunningAndGetDatumApplyPlayerControls(spProducer, spVideoSeek); + // Get Matrix vector + std::string nextFrameName = spProducer->getNextFrameName(); + const unsigned long long nextFrameNumber = datumProducerConstructorRunningAndGetNextFrameNumber( + spProducer); + const std::vector matrices = spProducer->getFrames(); // Check frames are not empty - checkIfTooManyConsecutiveEmptyFrames(mNumberConsecutiveEmptyFrames, cvMats.empty() || cvMats[0].empty()); - if (!cvMats.empty()) + checkIfTooManyConsecutiveEmptyFrames( + mNumberConsecutiveEmptyFrames, matrices.empty() || matrices[0].empty()); + if (!matrices.empty()) { - datums->resize(cvMats.size()); + // Get camera parameters + const std::vector cameraMatrices = spProducer->getCameraMatrices(); + const std::vector cameraExtrinsics = spProducer->getCameraExtrinsics(); + const std::vector cameraIntrinsics = spProducer->getCameraIntrinsics(); + // Resize datum + datums->resize(matrices.size()); // Datum cannot be assigned before resize() auto& datumPtr = (*datums)[0]; datumPtr = std::make_shared(); // Filling first element std::swap(datumPtr->name, nextFrameName); datumPtr->frameNumber = nextFrameNumber; - datumPtr->cvInputData = cvMats[0]; + datumPtr->cvInputData = matrices[0]; + datumProducerConstructorRunningAndGetDatumFrameIntegrity(datumPtr->cvInputData); if (!cameraMatrices.empty()) { datumPtr->cameraMatrix = cameraMatrices[0]; datumPtr->cameraExtrinsics = cameraExtrinsics[0]; datumPtr->cameraIntrinsics = cameraIntrinsics[0]; } - // Image integrity - if (datumPtr->cvInputData.channels() != 3) - { - const std::string commonMessage{"Input images must be 3-channel BGR."}; - // Grey to RGB if required - if (datumPtr->cvInputData.channels() == 1) - { - log(commonMessage + " Converting grey image into BGR.", Priority::High); - cv::cvtColor(datumPtr->cvInputData, datumPtr->cvInputData, CV_GRAY2BGR); - } - else - error(commonMessage, __LINE__, __FUNCTION__, __FILE__); - } + // Initially, cvOutputData = cvInputData. No performance hit (both cv::Mat share raw memory) datumPtr->cvOutputData = datumPtr->cvInputData; // Resize if it's stereo-system if (datums->size() > 1) { - // Stereo-system: Assign all cv::Mat + // Stereo-system: Assign all Matrices for (auto i = 1u ; i < datums->size() ; i++) { auto& datumIPtr = (*datums)[i]; datumIPtr = std::make_shared(); datumIPtr->name = datumPtr->name; datumIPtr->frameNumber = datumPtr->frameNumber; - datumIPtr->cvInputData = cvMats[i]; + datumIPtr->cvInputData = matrices[i]; + datumProducerConstructorRunningAndGetDatumFrameIntegrity(datumPtr->cvInputData); datumIPtr->cvOutputData = datumIPtr->cvInputData; if (cameraMatrices.size() > i) { @@ -184,7 +161,7 @@ namespace op } } // Check producer is running - if (!datumProducerRunning || (*datums)[0]->cvInputData.empty()) + if ((*datums)[0]->cvInputData.empty()) datums = nullptr; // Increase counter if successful image if (datums != nullptr) @@ -205,11 +182,8 @@ namespace op void DatumProducer::checkIfTooManyConsecutiveEmptyFrames( unsigned int& numberConsecutiveEmptyFrames, const bool emptyFrame) const { - numberConsecutiveEmptyFrames = (emptyFrame ? numberConsecutiveEmptyFrames+1 : 0); - const auto threshold = 500u; - if (numberConsecutiveEmptyFrames >= threshold) - error("Detected too many (" + std::to_string(numberConsecutiveEmptyFrames) + ") empty frames in a row.", - __LINE__, __FUNCTION__, __FILE__); + datumProducerConstructorTooManyConsecutiveEmptyFrames( + numberConsecutiveEmptyFrames, emptyFrame); } extern template class DatumProducer; diff --git a/include/openpose/producer/flirReader.hpp b/include/openpose/producer/flirReader.hpp index d228783a..04700895 100644 --- a/include/openpose/producer/flirReader.hpp +++ b/include/openpose/producer/flirReader.hpp @@ -23,11 +23,11 @@ namespace op virtual ~FlirReader(); - std::vector getCameraMatrices(); + std::vector getCameraMatrices(); - std::vector getCameraExtrinsics(); + std::vector getCameraExtrinsics(); - std::vector getCameraIntrinsics(); + std::vector getCameraIntrinsics(); std::string getNextFrameName(); @@ -44,9 +44,9 @@ namespace op Point mResolution; unsigned long long mFrameNameCounter; - cv::Mat getRawFrame(); + Matrix getRawFrame(); - std::vector getRawFrames(); + std::vector getRawFrames(); DELETE_COPY(FlirReader); }; diff --git a/include/openpose/producer/imageDirectoryReader.hpp b/include/openpose/producer/imageDirectoryReader.hpp index 40db9629..3a832687 100644 --- a/include/openpose/producer/imageDirectoryReader.hpp +++ b/include/openpose/producer/imageDirectoryReader.hpp @@ -51,9 +51,9 @@ namespace op Point mResolution; long long mFrameNameCounter; - cv::Mat getRawFrame(); + Matrix getRawFrame(); - std::vector getRawFrames(); + std::vector getRawFrames(); DELETE_COPY(ImageDirectoryReader); }; diff --git a/include/openpose/producer/ipCameraReader.hpp b/include/openpose/producer/ipCameraReader.hpp index 2b9c280a..47bd890b 100644 --- a/include/openpose/producer/ipCameraReader.hpp +++ b/include/openpose/producer/ipCameraReader.hpp @@ -41,9 +41,9 @@ namespace op private: const std::string mPathName; - cv::Mat getRawFrame(); + Matrix getRawFrame(); - std::vector getRawFrames(); + std::vector getRawFrames(); DELETE_COPY(IpCameraReader); }; diff --git a/include/openpose/producer/producer.hpp b/include/openpose/producer/producer.hpp index 114e9c7d..c00d43bf 100644 --- a/include/openpose/producer/producer.hpp +++ b/include/openpose/producer/producer.hpp @@ -1,8 +1,6 @@ #ifndef OPENPOSE_PRODUCER_PRODUCER_HPP #define OPENPOSE_PRODUCER_PRODUCER_HPP -#include // cv::Mat -#include // capProperties of OpenCV #include #include #include @@ -30,36 +28,36 @@ namespace op /** * Main function of Producer, it retrieves and returns a new frame from the frames producer. - * @return cv::Mat with the new frame. + * @return Mat with the new frame. */ - cv::Mat getFrame(); + Matrix getFrame(); /** * Analogous to getFrame, but it could return > 1 frame. - * @return std::vector with the new frame(s). + * @return std::vector with the new frame(s). */ - std::vector getFrames(); + std::vector getFrames(); /** * It retrieves and returns the camera matrixes from the frames producer. * Virtual class because FlirReader implements their own. - * @return std::vector with the camera matrices. + * @return std::vector with the camera matrices. */ - virtual std::vector getCameraMatrices(); + virtual std::vector getCameraMatrices(); /** * It retrieves and returns the camera extrinsic parameters from the frames producer. * Virtual class because FlirReader implements their own. - * @return std::vector with the camera extrinsic parameters. + * @return std::vector with the camera extrinsic parameters. */ - virtual std::vector getCameraExtrinsics(); + virtual std::vector getCameraExtrinsics(); /** * It retrieves and returns the camera intrinsic parameters from the frames producer. * Virtual class because FlirReader implements their own. - * @return std::vector with the camera intrinsic parameters. + * @return std::vector with the camera intrinsic parameters. */ - virtual std::vector getCameraIntrinsics(); + virtual std::vector getCameraIntrinsics(); /** * This function returns a unique frame name (e.g., the frame number for video, the @@ -132,10 +130,10 @@ namespace op /** * Protected function which checks that the frames keeps their integry (some OpenCV versions * might return corrupted frames within a video or webcam with a size different to the - * standard resolution). If the frame is corrupted, it is set to an empty cv::Mat. - * @param frame cv::Mat with the frame matrix to be checked and modified. + * standard resolution). If the frame is corrupted, it is set to an empty Mat. + * @param frame Mat with the frame matrix to be checked and modified. */ - void checkFrameIntegrity(cv::Mat& frame); + void checkFrameIntegrity(Matrix& frame); /** * Protected function which checks that the frame producer has ended. If so, if resets @@ -150,16 +148,16 @@ namespace op /** * Function to be defined by its children class. It retrieves and returns a new frame from the frames producer. - * @return cv::Mat with the new frame. + * @return Mat with the new frame. */ - virtual cv::Mat getRawFrame() = 0; + virtual Matrix getRawFrame() = 0; /** * Function to be defined by its children class. It retrieves and returns a new frame from the frames producer. * It is equivalent to getRawFrame when more than 1 image can be returned. - * @return std::vector with the new frames. + * @return std::vector with the new frames. */ - virtual std::vector getRawFrames() = 0; + virtual std::vector getRawFrames() = 0; private: const ProducerType mType; diff --git a/include/openpose/producer/spinnakerWrapper.hpp b/include/openpose/producer/spinnakerWrapper.hpp index a5623e2e..09fd09d8 100644 --- a/include/openpose/producer/spinnakerWrapper.hpp +++ b/include/openpose/producer/spinnakerWrapper.hpp @@ -21,17 +21,17 @@ namespace op virtual ~SpinnakerWrapper(); - std::vector getRawFrames(); + std::vector getRawFrames(); /** * Note: The camera parameters are only read if undistortImage is true. This should be changed to add a * new bool flag in the constructor, e.g., readCameraParameters */ - std::vector getCameraMatrices() const; + std::vector getCameraMatrices() const; - std::vector getCameraExtrinsics() const; + std::vector getCameraExtrinsics() const; - std::vector getCameraIntrinsics() const; + std::vector getCameraIntrinsics() const; Point getResolution() const; diff --git a/include/openpose/producer/videoCaptureReader.hpp b/include/openpose/producer/videoCaptureReader.hpp index 4dbbf8a8..d503d359 100644 --- a/include/openpose/producer/videoCaptureReader.hpp +++ b/include/openpose/producer/videoCaptureReader.hpp @@ -1,8 +1,6 @@ #ifndef OPENPOSE_PRODUCER_VIDEO_CAPTURE_READER_HPP #define OPENPOSE_PRODUCER_VIDEO_CAPTURE_READER_HPP -#include // cv::Mat -#include // cv::VideoCapture #include #include @@ -49,14 +47,17 @@ namespace op virtual void set(const int capProperty, const double value) = 0; protected: - virtual cv::Mat getRawFrame() = 0; + virtual Matrix getRawFrame() = 0; - virtual std::vector getRawFrames() = 0; + virtual std::vector getRawFrames() = 0; void resetWebcam(const int index, const bool throwExceptionIfNoOpened); private: - cv::VideoCapture mVideoCapture; + // PIMPL idiom + // http://www.cppsamples.com/common-tasks/pimpl.html + struct ImplVideoCaptureReader; + std::unique_ptr upImpl; DELETE_COPY(VideoCaptureReader); }; diff --git a/include/openpose/producer/videoReader.hpp b/include/openpose/producer/videoReader.hpp index 3e1162be..1132d056 100644 --- a/include/openpose/producer/videoReader.hpp +++ b/include/openpose/producer/videoReader.hpp @@ -42,9 +42,9 @@ namespace op private: const std::string mPathName; - cv::Mat getRawFrame(); + Matrix getRawFrame(); - std::vector getRawFrames(); + std::vector getRawFrames(); DELETE_COPY(VideoReader); }; diff --git a/include/openpose/producer/webcamReader.hpp b/include/openpose/producer/webcamReader.hpp index 9ad967b5..04b6f2c2 100644 --- a/include/openpose/producer/webcamReader.hpp +++ b/include/openpose/producer/webcamReader.hpp @@ -43,7 +43,7 @@ namespace op const bool mWebcamStarted; long long mFrameNameCounter; bool mThreadOpened; - cv::Mat mBuffer; + Matrix mBuffer; std::mutex mBufferMutex; std::atomic mCloseThread; std::thread mThread; @@ -52,9 +52,9 @@ namespace op std::atomic mDisconnectedCounter; Point mResolution; - cv::Mat getRawFrame(); + Matrix getRawFrame(); - std::vector getRawFrames(); + std::vector getRawFrames(); void bufferingThread(); diff --git a/include/openpose/thread/priorityQueue.hpp b/include/openpose/thread/priorityQueue.hpp index 4fbc283d..bac8caf9 100644 --- a/include/openpose/thread/priorityQueue.hpp +++ b/include/openpose/thread/priorityQueue.hpp @@ -1,5 +1,5 @@ #ifndef OPENPOSE_THREAD_PRIORITY_QUEUE_HPP -#define OPENPOSE_THREAD_PRIORITY_QUEUE_HPP +#define OPENPOSE_THREAD_PRIORITY_QUEUE_HPP #include // std::priority_queue #include diff --git a/include/openpose/thread/queue.hpp b/include/openpose/thread/queue.hpp index 8d7123d9..7b34a946 100644 --- a/include/openpose/thread/queue.hpp +++ b/include/openpose/thread/queue.hpp @@ -1,5 +1,5 @@ #ifndef OPENPOSE_THREAD_QUEUE_HPP -#define OPENPOSE_THREAD_QUEUE_HPP +#define OPENPOSE_THREAD_QUEUE_HPP #include // std::queue #include diff --git a/include/openpose/tracking/personIdExtractor.hpp b/include/openpose/tracking/personIdExtractor.hpp index 414bec42..78565d55 100644 --- a/include/openpose/tracking/personIdExtractor.hpp +++ b/include/openpose/tracking/personIdExtractor.hpp @@ -1,55 +1,30 @@ #ifndef OPENPOSE_TRACKING_PERSON_ID_EXTRACTOR_HPP #define OPENPOSE_TRACKING_PERSON_ID_EXTRACTOR_HPP -#include -#include -#include -#include #include namespace op { - struct PersonEntry - { - long long counterLastDetection; - std::vector keypoints; - std::vector status; - /* - PersonEntry(long long _last_frame, - std::vector _keypoints, - std::vector _active): - last_frame(_last_frame), keypoints(_keypoints), - active(_active) - {} - */ - }; class OP_API PersonIdExtractor { - public: PersonIdExtractor(const float confidenceThreshold = 0.1f, const float inlierRatioThreshold = 0.5f, const float distanceThreshold = 30.f, const int numberFramesToDeletePerson = 10); virtual ~PersonIdExtractor(); - Array extractIds(const Array& poseKeypoints, const cv::Mat& cvMatInput, + Array extractIds(const Array& poseKeypoints, const Matrix& cvMatInput, const unsigned long long imageViewIndex = 0ull); - Array extractIdsLockThread(const Array& poseKeypoints, const cv::Mat& cvMatInput, + Array extractIdsLockThread(const Array& poseKeypoints, const Matrix& cvMatInput, const unsigned long long imageViewIndex, const long long frameId); private: - const float mConfidenceThreshold; - const float mInlierRatioThreshold; - const float mDistanceThreshold; - const int mNumberFramesToDeletePerson; - long long mNextPersonId; - cv::Mat mImagePrevious; - std::vector mPyramidImagesPrevious; - std::unordered_map mPersonEntries; - // Thread-safe variables - std::atomic mLastFrameId; + // PIMPL idiom + // http://www.cppsamples.com/common-tasks/pimpl.html + struct ImplPersonIdExtractor; + std::shared_ptr spImpl; DELETE_COPY(PersonIdExtractor); }; diff --git a/include/openpose/tracking/personTracker.hpp b/include/openpose/tracking/personTracker.hpp index 75c3798a..33990188 100644 --- a/include/openpose/tracking/personTracker.hpp +++ b/include/openpose/tracking/personTracker.hpp @@ -1,34 +1,12 @@ -#ifndef OPENPOSE_TRACKING_PERSON_TRACKER_HPP -#define OPENPOSE_TRACKING_PERSON_TRACKER_HPP +#ifndef OPENPOSE_OPENPOSE_PRIVATE_TRACKING_PERSON_TRACKER_HPP +#define OPENPOSE_OPENPOSE_PRIVATE_TRACKING_PERSON_TRACKER_HPP -#include -#include #include namespace op { - struct PersonTrackerEntry - { - std::vector keypoints; - std::vector lastKeypoints; - std::vector status; - std::vector getPredicted() const - { - std::vector predictedKeypoints(keypoints); - if (!lastKeypoints.size()) - return predictedKeypoints; - for (size_t i=0; i& poseKeypoints, Array& poseIds, const cv::Mat& cvMatInput); + void track(Array& poseKeypoints, Array& poseIds, const Matrix& cvMatInput); - void trackLockThread(Array& poseKeypoints, Array& poseIds, const cv::Mat& cvMatInput, + void trackLockThread(Array& poseKeypoints, Array& poseIds, const Matrix& cvMatInput, const long long frameId); bool getMergeResults() const; private: - const bool mMergeResults; - const int mLevels; - const int mPatchSize; - const bool mTrackVelocity; - const float mConfidenceThreshold; - const bool mScaleVarying; - const float mRescale; - - cv::Mat mImagePrevious; - std::vector mPyramidImagesPrevious; - std::unordered_map mPersonEntries; - Array mLastPoseIds; - - // Thread-safe variables - std::atomic mLastFrameId; + // PIMPL idiom + // http://www.cppsamples.com/common-tasks/pimpl.html + struct ImplPersonTracker; + std::shared_ptr spImpl; DELETE_COPY(PersonTracker); }; } -#endif // OPENPOSE_TRACKING_PERSON_TRACKER_HPP +#endif // OPENPOSE_OPENPOSE_PRIVATE_TRACKING_PERSON_TRACKER_HPP diff --git a/include/openpose/tracking/pyramidalLK.hpp b/include/openpose/tracking/pyramidalLK.hpp deleted file mode 100644 index 4f8c839a..00000000 --- a/include/openpose/tracking/pyramidalLK.hpp +++ /dev/null @@ -1,26 +0,0 @@ -#ifndef OPENPOSE_TRACKING_LKPYRAMIDAL_HPP -#define OPENPOSE_TRACKING_LKPYRAMIDAL_HPP - -#include - -namespace op -{ - OP_API void pyramidalLKCpu(std::vector& coordI, std::vector& coordJ, - std::vector& pyramidImagesPrevious, - std::vector& pyramidImagesCurrent, - std::vector& status, const cv::Mat& imagePrevious, - const cv::Mat& imageCurrent, const int levels = 3, const int patchSize = 21); - - int pyramidalLKGpu(std::vector& ptsI, std::vector& ptsJ, - std::vector& status, const cv::Mat& imagePrevious, - const cv::Mat& imageCurrent, const int levels = 3, const int patchSize = 21); - - OP_API void pyramidalLKOcv(std::vector& coordI, std::vector& coordJ, - std::vector& pyramidImagesPrevious, - std::vector& pyramidImagesCurrent, - std::vector& status, const cv::Mat& imagePrevious, - const cv::Mat& imageCurrent, const int levels = 3, const int patchSize = 21, - const bool initFlow = false); -} - -#endif // OPENPOSE_TRACKING_LKPYRAMIDAL_HPP diff --git a/include/openpose/unity/unityBinding.hpp b/include/openpose/unity/unityBinding.hpp index 66b17e71..1a09ea81 100644 --- a/include/openpose/unity/unityBinding.hpp +++ b/include/openpose/unity/unityBinding.hpp @@ -1,3 +1,3 @@ -// Temporarily, all the code is located in +// Temporarily, all the code is located in // src/openpose/unity/unityBinding.cpp // TODO: Move functionality from unityBinding.cpp to this class diff --git a/include/openpose/utilities/openCv.hpp b/include/openpose/utilities/openCv.hpp index 816c20f5..b926929d 100644 --- a/include/openpose/utilities/openCv.hpp +++ b/include/openpose/utilities/openCv.hpp @@ -1,34 +1,65 @@ #ifndef OPENPOSE_UTILITIES_OPEN_CV_HPP #define OPENPOSE_UTILITIES_OPEN_CV_HPP -#include // cv::Mat -#include // cv::warpAffine, cv::BORDER_CONSTANT #include namespace op { - OP_API void putTextOnCvMat(cv::Mat& cvMat, const std::string& textToDisplay, const Point& position, - const cv::Scalar& color, const bool normalizeWidth, const int imageWidth); + OP_API void unrollArrayToUCharCvMat(Matrix& matResult, const Array& array); - OP_API void unrollArrayToUCharCvMat(cv::Mat& cvMatResult, const Array& array); - - OP_API void uCharCvMatToFloatPtr(float* floatPtrImage, const cv::Mat& cvImage, const int normalize); + OP_API void uCharCvMatToFloatPtr(float* floatPtrImage, const Matrix& matImage, const int normalize); OP_API double resizeGetScaleFactor(const Point& initialSize, const Point& targetSize); - OP_API void resizeFixedAspectRatio( - cv::Mat& resizedCvMat, const cv::Mat& cvMat, const double scaleFactor, const Point& targetSize, - const int borderMode = cv::BORDER_CONSTANT, const cv::Scalar& borderValue = cv::Scalar{0,0,0}); - - OP_API void keepRoiInside(cv::Rect& roi, const int imageWidth, const int imageHeight); + OP_API void keepRoiInside(Rectangle& roi, const int imageWidth, const int imageHeight); /** - * It performs rotation and flipping over the desired cv::Mat. - * @param cvMat cv::Mat with the frame matrix to be rotated and/or flipped. + * It performs rotation and flipping over the desired Mat. + * @param cvMat Mat with the frame matrix to be rotated and/or flipped. * @param rotationAngle How much the cvMat element should be rotated. 0 would mean no rotation. * @param flipFrame Whether to flip the cvMat element. Set to false to disable it. */ - OP_API void rotateAndFlipFrame(cv::Mat& cvMat, const double rotationAngle, const bool flipFrame = false); + OP_API void rotateAndFlipFrame(Matrix& cvMat, const double rotationAngle, const bool flipFrame = false); + + /** + * Wrapper of CV_CAP_PROP_FRAME_COUNT to avoid leaving OpenCV dependencies on headers. + */ + OP_API int getCvCapPropFrameCount(); + + /** + * Wrapper of CV_CAP_PROP_FRAME_FPS to avoid leaving OpenCV dependencies on headers. + */ + OP_API int getCvCapPropFrameFps(); + + /** + * Wrapper of CV_CAP_PROP_FRAME_WIDTH to avoid leaving OpenCV dependencies on headers. + */ + OP_API int getCvCapPropFrameWidth(); + + /** + * Wrapper of CV_CAP_PROP_FRAME_HEIGHT to avoid leaving OpenCV dependencies on headers. + */ + OP_API int getCvCapPropFrameHeight(); + + /** + * Wrapper of CV_FOURCC to avoid leaving OpenCV dependencies on headers. + */ + OP_API int getCvFourcc(const char c1, const char c2, const char c3, const char c4); + + /** + * Wrapper of CV_IMWRITE_JPEG_QUALITY to avoid leaving OpenCV dependencies on headers. + */ + OP_API int getCvImwriteJpegQuality(); + + /** + * Wrapper of CV_IMWRITE_PNG_COMPRESSION to avoid leaving OpenCV dependencies on headers. + */ + OP_API int getCvImwritePngCompression(); + + /** + * Wrapper of CV_LOAD_IMAGE_ANYDEPTH to avoid leaving OpenCV dependencies on headers. + */ + OP_API int getCvLoadImageAnydepth(); } #endif // OPENPOSE_UTILITIES_OPEN_CV_HPP diff --git a/include/openpose/wrapper/wrapper.hpp b/include/openpose/wrapper/wrapper.hpp index 1b38f52d..b3c6e41b 100644 --- a/include/openpose/wrapper/wrapper.hpp +++ b/include/openpose/wrapper/wrapper.hpp @@ -164,11 +164,11 @@ namespace op bool waitAndEmplace(TDatumsSP& tDatums); /** - * Similar to waitAndEmplace(const TDatumsSP& tDatums), but it takes a cv::Mat as input. - * @param cvMat cv::Mat with the image to be processed. + * Similar to waitAndEmplace(const TDatumsSP& tDatums), but it takes a Matrix as input. + * @param matrix Matrix with the image to be processed. * @return Boolean specifying whether the tDatums could be emplaced. */ - bool waitAndEmplace(cv::Mat& cvMat); + bool waitAndEmplace(Matrix& matrix); /** * Push (copy) an element on the first (input) queue. @@ -187,11 +187,11 @@ namespace op bool waitAndPush(const TDatumsSP& tDatums); /** - * Similar to waitAndPush(const TDatumsSP& tDatums), but it takes a cv::Mat as input. - * @param cvMat cv::Mat with the image to be processed. + * Similar to waitAndPush(const TDatumsSP& tDatums), but it takes a Matrix as input. + * @param matrix Matrix with the image to be processed. * @return Boolean specifying whether the tDatums could be pushed. */ - bool waitAndPush(const cv::Mat& cvMat); + bool waitAndPush(const Matrix& matrix); /** * Pop (retrieve) an element from the last (output) queue. @@ -220,11 +220,11 @@ namespace op bool emplaceAndPop(TDatumsSP& tDatums); /** - * Similar to emplaceAndPop(TDatumsSP& tDatums), but it takes a cv::Mat as input. - * @param cvMat cv::Mat with the image to be processed. + * Similar to emplaceAndPop(TDatumsSP& tDatums), but it takes a Matrix as input. + * @param matrix Matrix with the image to be processed. * @return TDatumsSP element where the processed information will be placed. */ - TDatumsSP emplaceAndPop(const cv::Mat& cvMat); + TDatumsSP emplaceAndPop(const Matrix& matrix); private: const ThreadManagerMode mThreadManagerMode; @@ -518,7 +518,7 @@ namespace op } template - bool WrapperT::waitAndEmplace(cv::Mat& cvMat) + bool WrapperT::waitAndEmplace(Matrix& matrix) { try { @@ -528,7 +528,7 @@ namespace op auto& tDatumPtr = datumsPtr->at(0); tDatumPtr = std::make_shared(); // Fill datum - std::swap(tDatumPtr->cvInputData, cvMat); + std::swap(tDatumPtr->cvInputData, matrix); // Return result return waitAndEmplace(datumsPtr); } @@ -574,7 +574,7 @@ namespace op } template - bool WrapperT::waitAndPush(const cv::Mat& cvMat) + bool WrapperT::waitAndPush(const Matrix& matrix) { try { @@ -584,7 +584,7 @@ namespace op auto& tDatumPtr = datumsPtr->at(0); tDatumPtr = std::make_shared(); // Fill datum - tDatumPtr->cvInputData = cvMat.clone(); + tDatumPtr->cvInputData = matrix.clone(); // Return result return waitAndEmplace(datumsPtr); } @@ -647,7 +647,7 @@ namespace op } template - TDatumsSP WrapperT::emplaceAndPop(const cv::Mat& cvMat) + TDatumsSP WrapperT::emplaceAndPop(const Matrix& matrix) { try { @@ -657,7 +657,7 @@ namespace op auto& tDatumPtr = datumsPtr->at(0); tDatumPtr = std::make_shared(); // Fill datum - tDatumPtr->cvInputData = cvMat; + tDatumPtr->cvInputData = matrix; // Emplace and pop emplaceAndPop(datumsPtr); // Return result diff --git a/include/openpose/wrapper/wrapperAuxiliary.hpp b/include/openpose/wrapper/wrapperAuxiliary.hpp index a1aa3e5f..a6e3ddc4 100644 --- a/include/openpose/wrapper/wrapperAuxiliary.hpp +++ b/include/openpose/wrapper/wrapperAuxiliary.hpp @@ -217,8 +217,8 @@ namespace op producerSharedPtr->set(ProducerProperty::Rotation, wrapperStructInput.frameRotate); producerSharedPtr->set(ProducerProperty::AutoRepeat, wrapperStructInput.framesRepeat); // 2. Set finalOutputSize - producerSize = Point{(int)producerSharedPtr->get(CV_CAP_PROP_FRAME_WIDTH), - (int)producerSharedPtr->get(CV_CAP_PROP_FRAME_HEIGHT)}; + producerSize = Point{(int)producerSharedPtr->get(getCvCapPropFrameWidth()), + (int)producerSharedPtr->get(getCvCapPropFrameHeight())}; // Set finalOutputSize to input size if desired if (finalOutputSize.x == -1 || finalOutputSize.y == -1) finalOutputSize = producerSize; @@ -708,7 +708,7 @@ namespace op { log("", Priority::Low, __LINE__, __FUNCTION__, __FILE__); const auto verbosePrinter = std::make_shared( - wrapperStructOutput.verbose, producerSharedPtr->get(CV_CAP_PROP_FRAME_COUNT)); + wrapperStructOutput.verbose, uLongLongRound(producerSharedPtr->get(getCvCapPropFrameCount()))); outputWs.emplace_back(std::make_shared>(verbosePrinter)); } log("", Priority::Low, __LINE__, __FUNCTION__, __FILE__); @@ -778,7 +778,7 @@ namespace op { log("", Priority::Low, __LINE__, __FUNCTION__, __FILE__); if (wrapperStructOutput.writeVideoFps <= 0 - && (!oPProducer || producerSharedPtr->get(CV_CAP_PROP_FPS) <= 0)) + && (!oPProducer || producerSharedPtr->get(getCvCapPropFrameFps()) <= 0)) error("The frame rate of the frames producer is unknown. Set `--write_video_fps` to your desired" " FPS if you wanna record video (`--write_video`). E.g., if it is a folder of images, you" " will have to know or guess the frame rate; if it is a webcam, you should use the OpenPose" @@ -786,7 +786,7 @@ namespace op __LINE__, __FUNCTION__, __FILE__); originalVideoFps = ( wrapperStructOutput.writeVideoFps > 0 ? - wrapperStructOutput.writeVideoFps : producerSharedPtr->get(CV_CAP_PROP_FPS)); + wrapperStructOutput.writeVideoFps : producerSharedPtr->get(getCvCapPropFrameFps())); } log("", Priority::Low, __LINE__, __FUNCTION__, __FILE__); // Write frames as *.avi video on hard disk @@ -804,7 +804,7 @@ namespace op __LINE__, __FUNCTION__, __FILE__); // Create video saver worker const auto videoSaver = std::make_shared( - wrapperStructOutput.writeVideo, CV_FOURCC('M','J','P','G'), originalVideoFps, + wrapperStructOutput.writeVideo, getCvFourcc('M','J','P','G'), originalVideoFps, (wrapperStructOutput.writeVideoWithAudio ? wrapperStructInput.producerString : "")); outputWs.emplace_back(std::make_shared>(videoSaver)); } @@ -900,7 +900,7 @@ namespace op if (!wrapperStructOutput.writeVideo3D.empty()) { const auto videoSaver = std::make_shared( - wrapperStructOutput.writeVideo3D, CV_FOURCC('M','J','P','G'), originalVideoFps, ""); + wrapperStructOutput.writeVideo3D, getCvFourcc('M','J','P','G'), originalVideoFps, ""); videoSaver3DW = std::make_shared>(videoSaver); } } diff --git a/include/openpose_private/3d/poseTriangulationPrivate.hpp b/include/openpose_private/3d/poseTriangulationPrivate.hpp new file mode 100644 index 00000000..7c0b5fff --- /dev/null +++ b/include/openpose_private/3d/poseTriangulationPrivate.hpp @@ -0,0 +1,28 @@ +#ifndef OPENPOSE_PRIVATE_3D_POSE_TRIANGULATION_PRIVATE_HPP +#define OPENPOSE_PRIVATE_3D_POSE_TRIANGULATION_PRIVATE_HPP + +#include +#include + +namespace op +{ + /** + * 3D triangulation given known camera parameter matrices and based on linear DLT algorithm. + * The returned cv::Mat is a 4x1 matrix, where the last coordinate is 1. + */ + double triangulate( + cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, + const std::vector& pointsOnEachCamera); + + /** + * 3D triangulation given known camera parameter matrices and based on linear DLT algorithm with additional LMA + * non-linear refinement. + * The returned cv::Mat is a 4x1 matrix, where the last coordinate is 1. + * Note: If Ceres is not enabled, the LMA refinement is skipped and this function is equivalent to triangulate(). + */ + double triangulateWithOptimization( + cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, + const std::vector& pointsOnEachCamera, const double reprojectionMaxAcceptable); +} + +#endif // OPENPOSE_PRIVATE_3D_POSE_TRIANGULATION_PRIVATE_HPP diff --git a/include/openpose/calibration/gridPatternFunctions.hpp b/include/openpose_private/calibration/gridPatternFunctions.hpp similarity index 52% rename from include/openpose/calibration/gridPatternFunctions.hpp rename to include/openpose_private/calibration/gridPatternFunctions.hpp index e9e7ff7d..ddf1724b 100644 --- a/include/openpose/calibration/gridPatternFunctions.hpp +++ b/include/openpose_private/calibration/gridPatternFunctions.hpp @@ -1,5 +1,5 @@ -#ifndef OPENPOSE_CALIBRATION_GRID_PATTERN_FUNCTIONS_HPP -#define OPENPOSE_CALIBRATION_GRID_PATTERN_FUNCTIONS_HPP +#ifndef OPENPOSE_PRIVATE_CALIBRATION_GRID_PATTERN_FUNCTIONS_HPP +#define OPENPOSE_PRIVATE_CALIBRATION_GRID_PATTERN_FUNCTIONS_HPP #include #include @@ -14,24 +14,25 @@ namespace op BottomRight }; - OP_API std::pair> findAccurateGridCorners( + std::pair> findAccurateGridCorners( const cv::Mat& image, const cv::Size& gridInnerCorners); - OP_API std::vector getObjects3DVector( + std::vector getObjects3DVector( const cv::Size& gridInnerCorners, const float gridSquareSizeMm); - OP_API void drawGridCorners( + void drawGridCorners( cv::Mat& image, const cv::Size& gridInnerCorners, const std::vector& points2DVector); - OP_API std::array getOutterCornerIndices( + std::array getOutterCornerIndices( const std::vector& points2DVector, const cv::Size& gridInnerCorners); - OP_API void reorderPoints(std::vector& points2DVector, const cv::Size& gridInnerCorners, - const cv::Mat& image, const bool showWarning = true); + void reorderPoints( + std::vector& points2DVector, const cv::Size& gridInnerCorners, + const cv::Mat& image, const bool showWarning = true); - OP_API void plotGridCorners( + void plotGridCorners( const cv::Size& gridInnerCorners, const std::vector& points2DVector, const std::string& imagePath, const cv::Mat& image); } -#endif // OPENPOSE_CALIBRATION_GRID_PATTERN_FUNCTIONS_HPP +#endif // OPENPOSE_PRIVATE_CALIBRATION_GRID_PATTERN_FUNCTIONS_HPP diff --git a/include/openpose/gpu/cl2.hpp b/include/openpose_private/gpu/cl2.hpp similarity index 97% rename from include/openpose/gpu/cl2.hpp rename to include/openpose_private/gpu/cl2.hpp index 0d6e805a..790c5982 100644 --- a/include/openpose/gpu/cl2.hpp +++ b/include/openpose_private/gpu/cl2.hpp @@ -84,17 +84,17 @@ * fixes in the new header as well as additional OpenCL 2.0 features. * As a result the header is not directly backward compatible and for this * reason we release it as cl2.hpp rather than a new version of cl.hpp. - * + * * * \section compatibility Compatibility * Due to the evolution of the underlying OpenCL API the 2.0 C++ bindings * include an updated approach to defining supported feature versions * and the range of valid underlying OpenCL runtime versions supported. * - * The combination of preprocessor macros CL_HPP_TARGET_OPENCL_VERSION and + * The combination of preprocessor macros CL_HPP_TARGET_OPENCL_VERSION and * CL_HPP_MINIMUM_OPENCL_VERSION control this range. These are three digit - * decimal values representing OpenCL runime versions. The default for - * the target is 200, representing OpenCL 2.0 and the minimum is also + * decimal values representing OpenCL runime versions. The default for + * the target is 200, representing OpenCL 2.0 and the minimum is also * defined as 200. These settings would use 2.0 API calls only. * If backward compatibility with a 1.2 runtime is required, the minimum * version may be set to 120. @@ -102,21 +102,21 @@ * Note that this is a compile-time setting, and so affects linking against * a particular SDK version rather than the versioning of the loaded runtime. * - * The earlier versions of the header included basic vector and string - * classes based loosely on STL versions. These were difficult to + * The earlier versions of the header included basic vector and string + * classes based loosely on STL versions. These were difficult to * maintain and very rarely used. For the 2.0 header we now assume * the presence of the standard library unless requested otherwise. - * We use std::array, std::vector, std::shared_ptr and std::string - * throughout to safely manage memory and reduce the chance of a + * We use std::array, std::vector, std::shared_ptr and std::string + * throughout to safely manage memory and reduce the chance of a * recurrance of earlier memory management bugs. * - * These classes are used through typedefs in the cl namespace: + * These classes are used through typedefs in the cl namespace: * cl::array, cl::vector, cl::pointer and cl::string. * In addition cl::allocate_pointer forwards to std::allocate_shared * by default. - * In all cases these standard library classes can be replaced with - * custom interface-compatible versions using the CL_HPP_NO_STD_ARRAY, - * CL_HPP_NO_STD_VECTOR, CL_HPP_NO_STD_UNIQUE_PTR and + * In all cases these standard library classes can be replaced with + * custom interface-compatible versions using the CL_HPP_NO_STD_ARRAY, + * CL_HPP_NO_STD_VECTOR, CL_HPP_NO_STD_UNIQUE_PTR and * CL_HPP_NO_STD_STRING macros. * * The OpenCL 1.x versions of the C++ bindings included a size_t wrapper @@ -127,12 +127,12 @@ * using the CL_HPP_ENABLE_SIZE_T_COMPATIBILITY macro. * * Finally, the program construction interface used a clumsy vector-of-pairs - * design in the earlier versions. We have replaced that with a cleaner - * vector-of-vectors and vector-of-strings design. However, for backward + * design in the earlier versions. We have replaced that with a cleaner + * vector-of-vectors and vector-of-strings design. However, for backward * compatibility old behaviour can be regained with the * CL_HPP_ENABLE_PROGRAM_CONSTRUCTION_FROM_ARRAY_COMPATIBILITY macro. - * - * In OpenCL 2.0 OpenCL C is not entirely backward compatibility with + * + * In OpenCL 2.0 OpenCL C is not entirely backward compatibility with * earlier versions. As a result a flag must be passed to the OpenCL C * compiled to request OpenCL 2.0 compilation of kernels with 1.2 as * the default in the absence of the flag. @@ -355,8 +355,8 @@ cl::unmapSVM(inputB); cl::unmapSVM(output2); - cl_int error; - vectorAddKernel( + cl_int error; + vectorAddKernel( cl::EnqueueArgs( cl::NDRange(numElements/2), cl::NDRange(numElements/2)), @@ -367,7 +367,7 @@ 3, aPipe, defaultDeviceQueue, - error + error ); cl::copy(outputBuffer, begin(output), end(output)); @@ -483,17 +483,17 @@ #if defined(_MSC_VER) #include -#endif // _MSC_VER - +#endif // _MSC_VER + // Check for a valid C++ version -// Need to do both tests here because for some reason __cplusplus is not +// Need to do both tests here because for some reason __cplusplus is not // updated in visual studio #if (!defined(_MSC_VER) && __cplusplus < 201103L) || (defined(_MSC_VER) && _MSC_VER < 1700) #error Visual studio 2013 or another C++11-supporting compiler required #endif -// +// #if defined(CL_HPP_USE_CL_DEVICE_FISSION) || defined(CL_HPP_USE_CL_SUB_GROUPS_KHR) #include #endif @@ -519,14 +519,14 @@ // Define deprecated prefixes and suffixes to ensure compilation // in case they are not pre-defined #if !defined(CL_EXT_PREFIX__VERSION_1_1_DEPRECATED) -#define CL_EXT_PREFIX__VERSION_1_1_DEPRECATED +#define CL_EXT_PREFIX__VERSION_1_1_DEPRECATED #endif // #if !defined(CL_EXT_PREFIX__VERSION_1_1_DEPRECATED) #if !defined(CL_EXT_SUFFIX__VERSION_1_1_DEPRECATED) #define CL_EXT_SUFFIX__VERSION_1_1_DEPRECATED #endif // #if !defined(CL_EXT_PREFIX__VERSION_1_1_DEPRECATED) #if !defined(CL_EXT_PREFIX__VERSION_1_2_DEPRECATED) -#define CL_EXT_PREFIX__VERSION_1_2_DEPRECATED +#define CL_EXT_PREFIX__VERSION_1_2_DEPRECATED #endif // #if !defined(CL_EXT_PREFIX__VERSION_1_2_DEPRECATED) #if !defined(CL_EXT_SUFFIX__VERSION_1_2_DEPRECATED) #define CL_EXT_SUFFIX__VERSION_1_2_DEPRECATED @@ -585,7 +585,7 @@ namespace cl { template using pointer = std::unique_ptr; } // namespace cl -#endif +#endif #endif // #if CL_HPP_TARGET_OPENCL_VERSION >= 200 #if !defined(CL_HPP_NO_STD_ARRAY) #include @@ -701,8 +701,8 @@ namespace cl { class Pipe; #if defined(CL_HPP_ENABLE_EXCEPTIONS) - /*! \brief Exception class - * + /*! \brief Exception class + * * This may be thrown by API functions when CL_HPP_ENABLE_EXCEPTIONS is defined. */ class Error : public std::exception @@ -713,7 +713,7 @@ namespace cl { public: /*! \brief Create a new CL error exception for a given error code * and corresponding message. - * + * * \param err error code value. * * \param errStr a descriptive string that must remain in scope until @@ -1062,7 +1062,7 @@ inline cl_int getInfoHelper(Func f, cl_uint name, array* param, lo if (err != CL_SUCCESS) { return err; } - + // Bound the copy with N to prevent overruns // if passed N > than the amount copied if (elements > N) { @@ -1467,7 +1467,7 @@ struct ReferenceHandler /** * Retain the device. * \param device A valid device created using createSubDevices - * \return + * \return * CL_SUCCESS if the function executed successfully. * CL_INVALID_DEVICE if device was not a valid subdevice * CL_OUT_OF_RESOURCES @@ -1478,7 +1478,7 @@ struct ReferenceHandler /** * Retain the device. * \param device A valid device created using createSubDevices - * \return + * \return * CL_SUCCESS if the function executed successfully. * CL_INVALID_DEVICE if device was not a valid subdevice * CL_OUT_OF_RESOURCES @@ -1641,11 +1641,11 @@ protected: public: Wrapper() : object_(NULL) { } - - Wrapper(const cl_type &obj, bool retainObject) : object_(obj) + + Wrapper(const cl_type &obj, bool retainObject) : object_(obj) { - if (retainObject) { - detail::errHandler(retain(), __RETAIN_ERR); + if (retainObject) { + detail::errHandler(retain(), __RETAIN_ERR); } } @@ -1756,15 +1756,15 @@ protected: } public: - Wrapper() : object_(NULL), referenceCountable_(false) - { - } - - Wrapper(const cl_type &obj, bool retainObject) : - object_(obj), - referenceCountable_(false) + Wrapper() : object_(NULL), referenceCountable_(false) { - referenceCountable_ = isReferenceCountable(obj); + } + + Wrapper(const cl_type &obj, bool retainObject) : + object_(obj), + referenceCountable_(false) + { + referenceCountable_ = isReferenceCountable(obj); if (retainObject) { detail::errHandler(retain(), __RETAIN_ERR); @@ -1775,11 +1775,11 @@ public: { release(); } - + Wrapper(const Wrapper& rhs) { object_ = rhs.object_; - referenceCountable_ = isReferenceCountable(object_); + referenceCountable_ = isReferenceCountable(object_); detail::errHandler(retain(), __RETAIN_ERR); } @@ -1818,7 +1818,7 @@ public: { detail::errHandler(release(), __RELEASE_ERR); object_ = rhs; - referenceCountable_ = isReferenceCountable(object_); + referenceCountable_ = isReferenceCountable(object_); return *this; } @@ -1996,10 +1996,10 @@ public: Device() : detail::Wrapper() { } /*! \brief Constructor from cl_device_id. - * + * * This simply copies the device ID value, which is an inexpensive operation. */ - explicit Device(const cl_device_id &device, bool retainObject = false) : + explicit Device(const cl_device_id &device, bool retainObject = false) : detail::Wrapper(device, retainObject) { } /*! \brief Returns the first device on the default context. @@ -2032,7 +2032,7 @@ public: } /*! \brief Assignment operator from cl_device_id. - * + * * This simply copies the device ID value, which is an inexpensive operation. */ Device& operator = (const cl_device_id& rhs) @@ -2113,7 +2113,7 @@ public: return detail::errHandler(err, __CREATE_SUB_DEVICES_ERR); } - // Cannot trivially assign because we need to capture intermediates + // Cannot trivially assign because we need to capture intermediates // with safe construction if (devices) { devices->resize(ids.size()); @@ -2121,7 +2121,7 @@ public: // Assign to param, constructing with retain behaviour // to correctly capture each underlying CL object for (size_type i = 0; i < ids.size(); i++) { - // We do not need to retain because this device is being created + // We do not need to retain because this device is being created // by the runtime (*devices)[i] = Device(ids[i], false); } @@ -2138,7 +2138,7 @@ public: const cl_device_partition_property_ext * properties, vector* devices) { - typedef CL_API_ENTRY cl_int + typedef CL_API_ENTRY cl_int ( CL_API_CALL * PFN_clCreateSubDevicesEXT)( cl_device_id /*in_device*/, const cl_device_partition_property_ext * /* properties */, @@ -2160,7 +2160,7 @@ public: if (err != CL_SUCCESS) { return detail::errHandler(err, __CREATE_SUB_DEVICES_ERR); } - // Cannot trivially assign because we need to capture intermediates + // Cannot trivially assign because we need to capture intermediates // with safe construction if (devices) { devices->resize(ids.size()); @@ -2168,7 +2168,7 @@ public: // Assign to param, constructing with retain behaviour // to correctly capture each underlying CL object for (size_type i = 0; i < ids.size(); i++) { - // We do not need to retain because this device is being created + // We do not need to retain because this device is being created // by the runtime (*devices)[i] = Device(ids[i], false); } @@ -2247,7 +2247,7 @@ private: static void makeDefaultProvided(const Platform &p) { default_ = p; } - + public: #ifdef CL_HPP_UNIT_TEST_ENABLE /*! \brief Reset the default. @@ -2265,17 +2265,17 @@ public: Platform() : detail::Wrapper() { } /*! \brief Constructor from cl_platform_id. - * + * * \param retainObject will cause the constructor to retain its cl object. * Defaults to false to maintain compatibility with * earlier versions. * This simply copies the platform ID value, which is an inexpensive operation. */ - explicit Platform(const cl_platform_id &platform, bool retainObject = false) : + explicit Platform(const cl_platform_id &platform, bool retainObject = false) : detail::Wrapper(platform, retainObject) { } /*! \brief Assignment operator from cl_platform_id. - * + * * This simply copies the platform ID value, which is an inexpensive operation. */ Platform& operator = (const cl_platform_id& rhs) @@ -2296,10 +2296,10 @@ public: } /** - * Modify the default platform to be used by + * Modify the default platform to be used by * subsequent operations. * Will only set the default if no default was previously created. - * @return updated default platform. + * @return updated default platform. * Should be compared to the passed value to ensure that it was updated. */ static Platform setDefault(const Platform &default_platform) @@ -2332,7 +2332,7 @@ public: } /*! \brief Gets a list of devices for this platform. - * + * * Wraps clGetDeviceIDs(). */ cl_int getDevices( @@ -2354,7 +2354,7 @@ public: return detail::errHandler(err, __GET_DEVICE_IDS_ERR); } - // Cannot trivially assign because we need to capture intermediates + // Cannot trivially assign because we need to capture intermediates // with safe construction // We must retain things we obtain from the API to avoid releasing // API-owned objects. @@ -2401,8 +2401,8 @@ public: vector* devices) const { typedef CL_API_ENTRY cl_int (CL_API_CALL *PFN_clGetDeviceIDsFromD3D10KHR)( - cl_platform_id platform, - cl_d3d10_device_source_khr d3d_device_source, + cl_platform_id platform, + cl_d3d10_device_source_khr d3d_device_source, void * d3d_object, cl_d3d10_device_set_khr d3d_device_set, cl_uint num_entries, @@ -2418,12 +2418,12 @@ public: cl_uint n = 0; cl_int err = pfn_clGetDeviceIDsFromD3D10KHR( - object_, - d3d_device_source, + object_, + d3d_device_source, d3d_object, - d3d_device_set, - 0, - NULL, + d3d_device_set, + 0, + NULL, &n); if (err != CL_SUCCESS) { return detail::errHandler(err, __GET_DEVICE_IDS_ERR); @@ -2431,18 +2431,18 @@ public: vector ids(n); err = pfn_clGetDeviceIDsFromD3D10KHR( - object_, - d3d_device_source, + object_, + d3d_device_source, d3d_object, d3d_device_set, - n, - ids.data(), + n, + ids.data(), NULL); if (err != CL_SUCCESS) { return detail::errHandler(err, __GET_DEVICE_IDS_ERR); } - // Cannot trivially assign because we need to capture intermediates + // Cannot trivially assign because we need to capture intermediates // with safe construction // We must retain things we obtain from the API to avoid releasing // API-owned objects. @@ -2460,7 +2460,7 @@ public: #endif /*! \brief Gets a list of available platforms. - * + * * Wraps clGetPlatformIDs(). */ static cl_int get( @@ -2495,7 +2495,7 @@ public: } /*! \brief Gets the first available platform. - * + * * Wraps clGetPlatformIDs(), returning the first result. */ static cl_int get( @@ -2526,8 +2526,8 @@ public: *errResult = err; } return default_platform; - } - + } + #if CL_HPP_TARGET_OPENCL_VERSION >= 120 //! \brief Wrapper for clUnloadCompiler(). cl_int @@ -2568,7 +2568,7 @@ UnloadCompiler() * * \see cl_context */ -class Context +class Context : public detail::Wrapper { private: @@ -2622,7 +2622,7 @@ private: static void makeDefaultProvided(const Context &c) { default_ = c; } - + public: #ifdef CL_HPP_UNIT_TEST_ENABLE /*! \brief Reset the default. @@ -2696,7 +2696,7 @@ public: *err = error; } } - + /*! \brief Constructs a context including all or a subset of devices of a specified type. * * Wraps clCreateContextFromType(). @@ -2815,7 +2815,7 @@ public: * * \note All calls to this function return the same cl_context as the first. */ - static Context getDefault(cl_int * err = NULL) + static Context getDefault(cl_int * err = NULL) { std::call_once(default_initialized_, makeDefault); detail::errHandler(default_error_); @@ -2843,15 +2843,15 @@ public: Context() : detail::Wrapper() { } /*! \brief Constructor from cl_context - takes ownership. - * + * * This effectively transfers ownership of a refcount on the cl_context * into the new Context object. */ - explicit Context(const cl_context& context, bool retainObject = false) : + explicit Context(const cl_context& context, bool retainObject = false) : detail::Wrapper(context, retainObject) { } /*! \brief Assignment operator from cl_context - takes ownership. - * + * * This effectively transfers ownership of a refcount on the rhs and calls * clReleaseContext() on the value previously held by this instance. */ @@ -2885,7 +2885,7 @@ public: } /*! \brief Gets a list of supported image formats. - * + * * Wraps clGetSupportedImageFormats(). */ cl_int getSupportedImageFormats( @@ -2894,17 +2894,17 @@ public: vector* formats) const { cl_uint numEntries; - + if (!formats) { return CL_SUCCESS; } cl_int err = ::clGetSupportedImageFormats( - object_, + object_, flags, - type, - 0, - NULL, + type, + 0, + NULL, &numEntries); if (err != CL_SUCCESS) { return detail::errHandler(err, __GET_SUPPORTED_IMAGE_FORMATS_ERR); @@ -2982,14 +2982,14 @@ public: Event() : detail::Wrapper() { } /*! \brief Constructor from cl_event - takes ownership. - * + * * \param retainObject will cause the constructor to retain its cl object. * Defaults to false to maintain compatibility with * earlier versions. * This effectively transfers ownership of a refcount on the cl_event * into the new Event object. */ - explicit Event(const cl_event& event, bool retainObject = false) : + explicit Event(const cl_event& event, bool retainObject = false) : detail::Wrapper(event, retainObject) { } /*! \brief Assignment operator from cl_event - takes ownership. @@ -3050,7 +3050,7 @@ public: } /*! \brief Blocks the calling thread until this event completes. - * + * * Wraps clWaitForEvents(). */ cl_int wait() const @@ -3067,7 +3067,7 @@ public: */ cl_int setCallback( cl_int type, - void (CL_CALLBACK * pfn_notify)(cl_event, cl_int, void *), + void (CL_CALLBACK * pfn_notify)(cl_event, cl_int, void *), void * user_data = NULL) { return detail::errHandler( @@ -3075,13 +3075,13 @@ public: object_, type, pfn_notify, - user_data), + user_data), __SET_EVENT_CALLBACK_ERR); } #endif // CL_HPP_TARGET_OPENCL_VERSION >= 110 /*! \brief Blocks the calling thread until every event specified is complete. - * + * * Wraps clWaitForEvents(). */ static cl_int @@ -3096,7 +3096,7 @@ public: #if CL_HPP_TARGET_OPENCL_VERSION >= 110 /*! \brief Class interface for user events (a subset of cl_event's). - * + * * See Event for details about copy semantics, etc. */ class UserEvent : public Event @@ -3131,14 +3131,14 @@ public: cl_int setStatus(cl_int status) { return detail::errHandler( - ::clSetUserEventStatus(object_,status), + ::clSetUserEventStatus(object_,status), __SET_USER_EVENT_STATUS_ERR); } }; #endif // CL_HPP_TARGET_OPENCL_VERSION >= 110 /*! \brief Blocks the calling thread until every event specified is complete. - * + * * Wraps clWaitForEvents(). */ inline static cl_int @@ -3256,14 +3256,14 @@ public: * value - not the Memory class instance. */ cl_int setDestructorCallback( - void (CL_CALLBACK * pfn_notify)(cl_mem, void *), + void (CL_CALLBACK * pfn_notify)(cl_mem, void *), void * user_data = NULL) { return detail::errHandler( ::clSetMemObjectDestructorCallback( object_, pfn_notify, - user_data), + user_data), __SET_MEM_OBJECT_DESTRUCTOR_CALLBACK_ERR); } #endif // CL_HPP_TARGET_OPENCL_VERSION >= 110 @@ -3446,7 +3446,7 @@ public: * Allocate an SVM pointer. * * If the allocator is coarse-grained, this will take ownership to allow - * containers to correctly construct data in place. + * containers to correctly construct data in place. */ pointer allocate( size_type size, @@ -3496,7 +3496,7 @@ public: for (Device &d : context_.getInfo()) { maxSize = std::min( - maxSize, + maxSize, static_cast(d.getInfo())); } @@ -3622,7 +3622,7 @@ cl::pointer>> allocate_svm(const cl #endif // #if !defined(CL_HPP_NO_STD_UNIQUE_PTR) /*! \brief Vector alias to simplify contruction of coarse-grained SVM containers. - * + * */ template < class T > using coarse_svm_vector = vector>>; @@ -3643,7 +3643,7 @@ using atomic_svm_vector = vector>> /*! \brief Class interface for Buffer Memory Objects. - * + * * See Memory for details about copy semantics, etc. * * \see Memory @@ -3728,7 +3728,7 @@ public: if( useHostPtr ) { flags |= CL_MEM_USE_HOST_PTR; } - + size_type size = sizeof(DataType)*(endIterator - startIterator); Context context = Context::getDefault(err); @@ -3761,7 +3761,7 @@ public: template< typename IteratorType > Buffer(const Context &context, IteratorType startIterator, IteratorType endIterator, bool readOnly, bool useHostPtr = false, cl_int* err = NULL); - + /*! * \brief Construct a Buffer from a host container via iterators using a specified queue. * If useHostPtr is specified iterators must be random access. @@ -3835,10 +3835,10 @@ public: Buffer result; cl_int error; result.object_ = ::clCreateSubBuffer( - object_, - flags, - buffer_create_type, - buffer_create_info, + object_, + flags, + buffer_create_type, + buffer_create_info, &error); detail::errHandler(error, __CREATE_SUBBUFFER_ERR); @@ -3847,7 +3847,7 @@ public: } return result; - } + } #endif // CL_HPP_TARGET_OPENCL_VERSION >= 110 }; @@ -3855,7 +3855,7 @@ public: /*! \brief Class interface for creating OpenCL buffers from ID3D10Buffer's. * * This is provided to facilitate interoperability with Direct3D. - * + * * See Memory for details about copy semantics, etc. * * \see Memory @@ -3863,7 +3863,7 @@ public: class BufferD3D10 : public Buffer { public: - + /*! \brief Constructs a BufferD3D10, in a specified context, from a * given ID3D10Buffer. @@ -3912,11 +3912,11 @@ public: /*! \brief Constructor from cl_mem - takes ownership. * * \param retainObject will cause the constructor to retain its cl object. - * Defaults to false to maintain compatibility with + * Defaults to false to maintain compatibility with * earlier versions. * See Memory for further details. */ - explicit BufferD3D10(const cl_mem& buffer, bool retainObject = false) : + explicit BufferD3D10(const cl_mem& buffer, bool retainObject = false) : Buffer(buffer, retainObject) { } /*! \brief Assignment from cl_mem - performs shallow copy. @@ -3932,7 +3932,7 @@ public: /*! \brief Copy constructor to forward copy to the superclass correctly. * Required for MSVC. */ - BufferD3D10(const BufferD3D10& buf) : + BufferD3D10(const BufferD3D10& buf) : Buffer(buf) {} /*! \brief Copy assignment to forward copy to the superclass correctly. @@ -3963,9 +3963,9 @@ public: /*! \brief Class interface for GL Buffer Memory Objects. * * This is provided to facilitate interoperability with OpenGL. - * + * * See Memory for details about copy semantics, etc. - * + * * \see Memory */ class BufferGL : public Buffer @@ -4060,9 +4060,9 @@ public: /*! \brief Class interface for GL Render Buffer Memory Objects. * * This is provided to facilitate interoperability with OpenGL. - * + * * See Memory for details about copy semantics, etc. - * + * * \see Memory */ class BufferRenderGL : public Buffer @@ -4098,7 +4098,7 @@ public: /*! \brief Constructor from cl_mem - takes ownership. * * \param retainObject will cause the constructor to retain its cl object. - * Defaults to false to maintain compatibility with + * Defaults to false to maintain compatibility with * earlier versions. * See Memory for further details. */ @@ -4157,7 +4157,7 @@ public: /*! \brief C++ base class for Image Memory objects. * * See Memory for details about copy semantics, etc. - * + * * \see Memory */ class Image : public Memory @@ -4224,7 +4224,7 @@ public: detail::getInfo(&::clGetImageInfo, object_, name, param), __GET_IMAGE_INFO_ERR); } - + //! \brief Wrapper for clGetImageInfo() that returns by value. template typename detail::param_traits::param_type @@ -4244,7 +4244,7 @@ public: /*! \brief Class interface for 1D Image Memory objects. * * See Memory for details about copy semantics, etc. - * + * * \see Memory */ class Image1D : public Image @@ -4270,11 +4270,11 @@ public: 0, 0, 0, 0, 0, 0, 0, 0 }; object_ = ::clCreateImage( - context(), - flags, - &format, - &desc, - host_ptr, + context(), + flags, + &format, + &desc, + host_ptr, &error); detail::errHandler(error, __CREATE_IMAGE_ERR); @@ -4359,11 +4359,11 @@ public: buffer() }; object_ = ::clCreateImage( - context(), - flags, - &format, - &desc, - NULL, + context(), + flags, + &format, + &desc, + NULL, &error); detail::errHandler(error, __CREATE_IMAGE_ERR); @@ -4447,11 +4447,11 @@ public: 0, 0, 0, 0 }; object_ = ::clCreateImage( - context(), - flags, - &format, - &desc, - host_ptr, + context(), + flags, + &format, + &desc, + host_ptr, &error); detail::errHandler(error, __CREATE_IMAGE_ERR); @@ -4461,7 +4461,7 @@ public: } Image1DArray() { } - + /*! \brief Constructor from cl_mem - takes ownership. * * \param retainObject will cause the constructor to retain its cl object. @@ -4514,7 +4514,7 @@ public: /*! \brief Class interface for 2D Image Memory objects. * * See Memory for details about copy semantics, etc. - * + * * \see Memory */ class Image2D : public Image @@ -4637,10 +4637,10 @@ public: * \note This will share storage with the underlying image but may * reinterpret the channel order and type. * - * The image will be created matching with a descriptor matching the source. + * The image will be created matching with a descriptor matching the source. * * \param order is the channel order to reinterpret the image data as. - * The channel order may differ as described in the OpenCL + * The channel order may differ as described in the OpenCL * 2.0 API specification. * * Wraps clCreateImage(). @@ -4654,9 +4654,9 @@ public: cl_int error; // Descriptor fields have to match source image - size_type sourceWidth = + size_type sourceWidth = sourceImage.getImageInfo(); - size_type sourceHeight = + size_type sourceHeight = sourceImage.getImageInfo(); size_type sourceRowPitch = sourceImage.getImageInfo(); @@ -4667,7 +4667,7 @@ public: cl_image_format sourceFormat = sourceImage.getImageInfo(); - // Update only the channel order. + // Update only the channel order. // Channel format inherited from source. sourceFormat.image_channel_order = order; cl_image_desc desc = @@ -4756,13 +4756,13 @@ public: /*! \brief Class interface for GL 2D Image Memory objects. * * This is provided to facilitate interoperability with OpenGL. - * + * * See Memory for details about copy semantics, etc. - * + * * \see Memory * \note Deprecated for OpenCL 1.2. Please use ImageGL instead. */ -class CL_EXT_PREFIX__VERSION_1_1_DEPRECATED Image2DGL : public Image2D +class CL_EXT_PREFIX__VERSION_1_1_DEPRECATED Image2DGL : public Image2D { public: /*! \brief Constructs an Image2DGL in a specified context, from a given @@ -4793,7 +4793,7 @@ public: } } - + //! \brief Default constructor - initializes to NULL. Image2DGL() : Image2D() { } @@ -4804,7 +4804,7 @@ public: * earlier versions. * See Memory for further details. */ - explicit Image2DGL(const cl_mem& image, bool retainObject = false) : + explicit Image2DGL(const cl_mem& image, bool retainObject = false) : Image2D(image, retainObject) { } /*! \brief Assignment from cl_mem - performs shallow copy. @@ -4880,11 +4880,11 @@ public: 0, 0, 0 }; object_ = ::clCreateImage( - context(), - flags, - &format, - &desc, - host_ptr, + context(), + flags, + &format, + &desc, + host_ptr, &error); detail::errHandler(error, __CREATE_IMAGE_ERR); @@ -4894,7 +4894,7 @@ public: } Image2DArray() { } - + /*! \brief Constructor from cl_mem - takes ownership. * * \param retainObject will cause the constructor to retain its cl object. @@ -4943,7 +4943,7 @@ public: /*! \brief Class interface for 3D Image Memory objects. * * See Memory for details about copy semantics, etc. - * + * * \see Memory */ class Image3D : public Image @@ -4995,11 +4995,11 @@ public: 0, 0, 0 }; object_ = ::clCreateImage( - context(), - flags, - &format, - &desc, - host_ptr, + context(), + flags, + &format, + &desc, + host_ptr, &error); detail::errHandler(error, __CREATE_IMAGE_ERR); @@ -5033,7 +5033,7 @@ public: * earlier versions. * See Memory for further details. */ - explicit Image3D(const cl_mem& image3D, bool retainObject = false) : + explicit Image3D(const cl_mem& image3D, bool retainObject = false) : Image(image3D, retainObject) { } /*! \brief Assignment from cl_mem - performs shallow copy. @@ -5079,9 +5079,9 @@ public: /*! \brief Class interface for GL 3D Image Memory objects. * * This is provided to facilitate interoperability with OpenGL. - * + * * See Memory for details about copy semantics, etc. - * + * * \see Memory */ class Image3DGL : public Image3D @@ -5125,7 +5125,7 @@ public: * earlier versions. * See Memory for further details. */ - explicit Image3DGL(const cl_mem& image, bool retainObject = false) : + explicit Image3DGL(const cl_mem& image, bool retainObject = false) : Image3D(image, retainObject) { } /*! \brief Assignment from cl_mem - performs shallow copy. @@ -5188,8 +5188,8 @@ public: { cl_int error; object_ = ::clCreateFromGLTexture( - context(), - flags, + context(), + flags, target, miplevel, texobj, @@ -5202,7 +5202,7 @@ public: } ImageGL() : Image() { } - + /*! \brief Constructor from cl_mem - takes ownership. * * \param retainObject will cause the constructor to retain its cl object. @@ -5210,7 +5210,7 @@ public: * earlier versions. * See Memory for further details. */ - explicit ImageGL(const cl_mem& image, bool retainObject = false) : + explicit ImageGL(const cl_mem& image, bool retainObject = false) : Image(image, retainObject) { } ImageGL& operator = (const cl_mem& rhs) @@ -5397,7 +5397,7 @@ public: * to the same underlying cl_sampler as the original. For details, see * clRetainSampler() and clReleaseSampler(). * - * \see cl_sampler + * \see cl_sampler */ class Sampler : public detail::Wrapper { @@ -5445,18 +5445,18 @@ public: if (err != NULL) { *err = error; } -#endif +#endif } /*! \brief Constructor from cl_sampler - takes ownership. - * + * * \param retainObject will cause the constructor to retain its cl object. * Defaults to false to maintain compatibility with * earlier versions. * This effectively transfers ownership of a refcount on the cl_sampler * into the new Sampler object. */ - explicit Sampler(const cl_sampler& sampler, bool retainObject = false) : + explicit Sampler(const cl_sampler& sampler, bool retainObject = false) : detail::Wrapper(sampler, retainObject) { } /*! \brief Assignment operator from cl_sampler - takes ownership. @@ -5572,17 +5572,17 @@ public: } /*! \brief Conversion operator to const size_type *. - * + * * \returns a pointer to the size of the first dimension. */ - operator const size_type*() const { - return sizes_; + operator const size_type*() const { + return sizes_; } //! \brief Queries the number of dimensions in the range. - size_type dimensions() const - { - return dimensions_; + size_type dimensions() const + { + return dimensions_; } //! \brief Returns the size of the object in bytes based on the @@ -5596,7 +5596,7 @@ public: { return sizes_; } - + const size_type* get() const { return sizes_; @@ -5644,7 +5644,7 @@ struct KernelArgumentHandler static const void* ptr(const LocalSpaceArg&) { return NULL; } }; -} +} //! \endcond /*! Local @@ -5674,14 +5674,14 @@ public: Kernel() { } /*! \brief Constructor from cl_kernel - takes ownership. - * + * * \param retainObject will cause the constructor to retain its cl object. * Defaults to false to maintain compatibility with * earlier versions. * This effectively transfers ownership of a refcount on the cl_kernel * into the new Kernel object. */ - explicit Kernel(const cl_kernel& kernel, bool retainObject = false) : + explicit Kernel(const cl_kernel& kernel, bool retainObject = false) : detail::Wrapper(kernel, retainObject) { } /*! \brief Assignment operator from cl_kernel - takes ownership. @@ -5789,7 +5789,7 @@ public: } return param; } - + #if CL_HPP_TARGET_OPENCL_VERSION >= 200 #if defined(CL_HPP_USE_CL_SUB_GROUPS_KHR) cl_int getSubGroupInfo(const cl::Device &dev, cl_kernel_sub_group_info name, const cl::NDRange &range, size_type* param) const @@ -5873,7 +5873,7 @@ public: #if CL_HPP_TARGET_OPENCL_VERSION >= 200 /*! - * Specify a vector of SVM pointers that the kernel may access in + * Specify a vector of SVM pointers that the kernel may access in * addition to its arguments. */ cl_int setSVMPointers(const vector &pointerList) @@ -5905,7 +5905,7 @@ public: * * \note It is only possible to enable fine-grained system SVM if all devices * in the context associated with kernel support it. - * + * * \param svmEnabled True if fine-grained system SVM is requested. False otherwise. * \return CL_SUCCESS if the function was executed succesfully. CL_INVALID_OPERATION * if no devices in the context support fine-grained system SVM. @@ -5924,7 +5924,7 @@ public: ) ); } - + template void setSVMPointersHelper(std::array &pointerList, const pointer &t0, Ts... ts) { @@ -5939,7 +5939,7 @@ public: pointerList[index] = static_cast(t0); setSVMPointersHelper(ts...); } - + template void setSVMPointersHelper(std::array &pointerList, const pointer &t0) { @@ -5982,7 +5982,7 @@ public: typedef vector > Binaries; typedef vector > Sources; #endif // #if !defined(CL_HPP_ENABLE_PROGRAM_CONSTRUCTION_FROM_ARRAY_COMPATIBILITY) - + Program( const string& source, bool build = false, @@ -6050,7 +6050,7 @@ public: #endif // #if !defined(CL_HPP_CL_1_2_DEFAULT_BUILD) NULL, NULL); - + detail::buildErrHandler(error, __BUILD_PROGRAM_ERR, getBuildInfo()); } @@ -6142,7 +6142,7 @@ public: * Set to CL_INVALID_BINARY if the binary provided is not valid for the matching device. * \param err if non-NULL will be set to CL_SUCCESS on successful operation or one of the following errors: * CL_INVALID_CONTEXT if context is not a valid context. - * CL_INVALID_VALUE if the length of devices is zero; or if the length of binaries does not match the length of devices; + * CL_INVALID_VALUE if the length of devices is zero; or if the length of binaries does not match the length of devices; * or if any entry in binaries is NULL or has length 0. * CL_INVALID_DEVICE if OpenCL devices listed in devices are not in the list of devices associated with context. * CL_INVALID_BINARY if an invalid program binary was encountered for any device. binaryStatus will return specific status for each device. @@ -6156,9 +6156,9 @@ public: cl_int* err = NULL) { cl_int error; - + const size_type numDevices = devices.size(); - + // Catch size mismatch early and return if(binaries.size() != numDevices) { error = CL_INVALID_VALUE; @@ -6183,7 +6183,7 @@ public: lengths[i] = binaries[(int)i].second; } #endif // #if !defined(CL_HPP_ENABLE_PROGRAM_CONSTRUCTION_FROM_ARRAY_COMPATIBILITY) - + vector deviceIDs(numDevices); for( size_type deviceIndex = 0; deviceIndex < numDevices; ++deviceIndex ) { deviceIDs[deviceIndex] = (devices[deviceIndex])(); @@ -6192,7 +6192,7 @@ public: if(binaryStatus) { binaryStatus->resize(numDevices); } - + object_ = ::clCreateProgramWithBinary( context(), (cl_uint) devices.size(), deviceIDs.data(), @@ -6206,7 +6206,7 @@ public: } } - + #if CL_HPP_TARGET_OPENCL_VERSION >= 120 /** * Create program using builtin kernels. @@ -6226,12 +6226,12 @@ public: for( size_type deviceIndex = 0; deviceIndex < numDevices; ++deviceIndex ) { deviceIDs[deviceIndex] = (devices[deviceIndex])(); } - + object_ = ::clCreateProgramWithBuiltInKernels( - context(), + context(), (cl_uint) devices.size(), deviceIDs.data(), - kernelNames.c_str(), + kernelNames.c_str(), &error); detail::errHandler(error, __CREATE_PROGRAM_WITH_BUILT_IN_KERNELS_ERR); @@ -6242,7 +6242,7 @@ public: #endif // CL_HPP_TARGET_OPENCL_VERSION >= 120 Program() { } - + /*! \brief Constructor from cl_mem - takes ownership. * @@ -6250,7 +6250,7 @@ public: * Defaults to false to maintain compatibility with * earlier versions. */ - explicit Program(const cl_program& program, bool retainObject = false) : + explicit Program(const cl_program& program, bool retainObject = false) : detail::Wrapper(program, retainObject) { } Program& operator = (const cl_program& rhs) @@ -6295,7 +6295,7 @@ public: { size_type numDevices = devices.size(); vector deviceIDs(numDevices); - + for( size_type deviceIndex = 0; deviceIndex < numDevices; ++deviceIndex ) { deviceIDs[deviceIndex] = (devices[deviceIndex])(); } @@ -6392,9 +6392,9 @@ public: } return param; } - + /** - * Build info function that returns a vector of device/info pairs for the specified + * Build info function that returns a vector of device/info pairs for the specified * info type and for all devices in the program. * On an error reading the info for any device, an empty vector of info will be returned. */ @@ -6446,7 +6446,7 @@ public: } vector value(numKernels); - + err = ::clCreateKernelsInProgram( object_, numKernels, value.data(), NULL); if (err != CL_SUCCESS) { @@ -6459,7 +6459,7 @@ public: // Assign to param, constructing with retain behaviour // to correctly capture each underlying CL object for (size_type i = 0; i < value.size(); i++) { - // We do not need to retain because this kernel is being created + // We do not need to retain because this kernel is being created // by the runtime (*kernels)[i] = Kernel(value[i], false); } @@ -6475,7 +6475,7 @@ inline Program linkProgram( const char* options = NULL, void (CL_CALLBACK * notifyFptr)(cl_program, void *) = NULL, void* data = NULL, - cl_int* err = NULL) + cl_int* err = NULL) { cl_int error_local = CL_SUCCESS; @@ -6510,7 +6510,7 @@ inline Program linkProgram( const char* options = NULL, void (CL_CALLBACK * notifyFptr)(cl_program, void *) = NULL, void* data = NULL, - cl_int* err = NULL) + cl_int* err = NULL) { cl_int error_local = CL_SUCCESS; @@ -6519,7 +6519,7 @@ inline Program linkProgram( for (unsigned int i = 0; i < inputPrograms.size(); i++) { programs[i] = inputPrograms[i](); } - + Context ctx; if(inputPrograms.size() > 0) { ctx = inputPrograms[0].getInfo(&error_local); @@ -6675,7 +6675,7 @@ public: default_ = CommandQueue(); } #endif // #ifdef CL_HPP_UNIT_TEST_ENABLE - + /*! * \brief Constructs a CommandQueue based on passed properties. @@ -6848,7 +6848,7 @@ public: CL_QUEUE_PROPERTIES, static_cast(properties), 0 }; object_ = ::clCreateCommandQueueWithProperties( context(), devices[0](), queue_properties, &error); - + detail::errHandler(error, __CREATE_COMMAND_QUEUE_WITH_PROPERTIES_ERR); if (err != NULL) { *err = error; @@ -6882,7 +6882,7 @@ public: CL_QUEUE_PROPERTIES, properties, 0 }; object_ = ::clCreateCommandQueueWithProperties( context(), device(), queue_properties, &error); - + detail::errHandler(error, __CREATE_COMMAND_QUEUE_WITH_PROPERTIES_ERR); if (err != NULL) { *err = error; @@ -6915,7 +6915,7 @@ public: CL_QUEUE_PROPERTIES, static_cast(properties), 0 }; object_ = ::clCreateCommandQueueWithProperties( context(), device(), queue_properties, &error); - + detail::errHandler(error, __CREATE_COMMAND_QUEUE_WITH_PROPERTIES_ERR); if (err != NULL) { *err = error; @@ -6931,7 +6931,7 @@ public: #endif } - static CommandQueue getDefault(cl_int * err = NULL) + static CommandQueue getDefault(cl_int * err = NULL) { std::call_once(default_initialized_, makeDefault); #if CL_HPP_TARGET_OPENCL_VERSION >= 200 @@ -6968,7 +6968,7 @@ public: * Defaults to false to maintain compatibility with * earlier versions. */ - explicit CommandQueue(const cl_command_queue& commandQueue, bool retainObject = false) : + explicit CommandQueue(const cl_command_queue& commandQueue, bool retainObject = false) : detail::Wrapper(commandQueue, retainObject) { } CommandQueue& operator = (const cl_command_queue& rhs) @@ -7118,8 +7118,8 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueReadBufferRect( - object_, - buffer(), + object_, + buffer(), blocking, buffer_offset.data(), host_offset.data(), @@ -7157,8 +7157,8 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueWriteBufferRect( - object_, - buffer(), + object_, + buffer(), blocking, buffer_offset.data(), host_offset.data(), @@ -7195,9 +7195,9 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueCopyBufferRect( - object_, - src(), - dst(), + object_, + src(), + dst(), src_origin.data(), dst_origin.data(), region.data(), @@ -7220,10 +7220,10 @@ public: /** * Enqueue a command to fill a buffer object with a pattern * of a given size. The pattern is specified as a vector type. - * \tparam PatternType The datatype of the pattern field. + * \tparam PatternType The datatype of the pattern field. * The pattern type must be an accepted OpenCL data type. - * \tparam offset Is the offset in bytes into the buffer at - * which to start filling. This must be a multiple of + * \tparam offset Is the offset in bytes into the buffer at + * which to start filling. This must be a multiple of * the pattern size. * \tparam size Is the size in bytes of the region to fill. * This must be a multiple of the pattern size. @@ -7240,11 +7240,11 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueFillBuffer( - object_, + object_, buffer(), static_cast(&pattern), - sizeof(PatternType), - offset, + sizeof(PatternType), + offset, size, (events != NULL) ? (cl_uint) events->size() : 0, (events != NULL && events->size() > 0) ? (cl_event*) &events->front() : NULL, @@ -7272,13 +7272,13 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueReadImage( - object_, - image(), - blocking, + object_, + image(), + blocking, origin.data(), - region.data(), - row_pitch, - slice_pitch, + region.data(), + row_pitch, + slice_pitch, ptr, (events != NULL) ? (cl_uint) events->size() : 0, (events != NULL && events->size() > 0) ? (cl_event*) &events->front() : NULL, @@ -7305,13 +7305,13 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueWriteImage( - object_, - image(), - blocking, + object_, + image(), + blocking, origin.data(), - region.data(), - row_pitch, - slice_pitch, + region.data(), + row_pitch, + slice_pitch, ptr, (events != NULL) ? (cl_uint) events->size() : 0, (events != NULL && events->size() > 0) ? (cl_event*) &events->front() : NULL, @@ -7336,11 +7336,11 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueCopyImage( - object_, - src(), - dst(), + object_, + src(), + dst(), src_origin.data(), - dst_origin.data(), + dst_origin.data(), region.data(), (events != NULL) ? (cl_uint) events->size() : 0, (events != NULL && events->size() > 0) ? (cl_event*) &events->front() : NULL, @@ -7372,9 +7372,9 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueFillImage( - object_, + object_, image(), - static_cast(&fillColor), + static_cast(&fillColor), origin.data(), region.data(), (events != NULL) ? (cl_uint) events->size() : 0, @@ -7406,9 +7406,9 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueFillImage( - object_, + object_, image(), - static_cast(&fillColor), + static_cast(&fillColor), origin.data(), region.data(), (events != NULL) ? (cl_uint) events->size() : 0, @@ -7440,9 +7440,9 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueFillImage( - object_, + object_, image(), - static_cast(&fillColor), + static_cast(&fillColor), origin.data(), region.data(), (events != NULL) ? (cl_uint) events->size() : 0, @@ -7469,11 +7469,11 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueCopyImageToBuffer( - object_, - src(), - dst(), + object_, + src(), + dst(), src_origin.data(), - region.data(), + region.data(), dst_offset, (events != NULL) ? (cl_uint) events->size() : 0, (events != NULL && events->size() > 0) ? (cl_event*) &events->front() : NULL, @@ -7498,11 +7498,11 @@ public: cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueCopyBufferToImage( - object_, - src(), - dst(), + object_, + src(), + dst(), src_offset, - dst_origin.data(), + dst_origin.data(), region.data(), (events != NULL) ? (cl_uint) events->size() : 0, (events != NULL && events->size() > 0) ? (cl_event*) &events->front() : NULL, @@ -7560,7 +7560,7 @@ public: cl_int error; void * result = ::clEnqueueMapImage( object_, buffer(), blocking, flags, - origin.data(), + origin.data(), region.data(), row_pitch, slice_pitch, (events != NULL) ? (cl_uint) events->size() : 0, @@ -7761,14 +7761,14 @@ public: #if CL_HPP_TARGET_OPENCL_VERSION >= 120 /** - * Enqueues a marker command which waits for either a list of events to complete, + * Enqueues a marker command which waits for either a list of events to complete, * or all previously enqueued commands to complete. * - * Enqueues a marker command which waits for either a list of events to complete, - * or if the list is empty it waits for all commands previously enqueued in command_queue - * to complete before it completes. This command returns an event which can be waited on, - * i.e. this event can be waited on to insure that all events either in the event_wait_list - * or all previously enqueued commands, queued before this command to command_queue, + * Enqueues a marker command which waits for either a list of events to complete, + * or if the list is empty it waits for all commands previously enqueued in command_queue + * to complete before it completes. This command returns an event which can be waited on, + * i.e. this event can be waited on to insure that all events either in the event_wait_list + * or all previously enqueued commands, queued before this command to command_queue, * have completed. */ cl_int enqueueMarkerWithWaitList( @@ -7793,12 +7793,12 @@ public: /** * A synchronization point that enqueues a barrier operation. * - * Enqueues a barrier command which waits for either a list of events to complete, - * or if the list is empty it waits for all commands previously enqueued in command_queue - * to complete before it completes. This command blocks command execution, that is, any - * following commands enqueued after it do not execute until it completes. This command - * returns an event which can be waited on, i.e. this event can be waited on to insure that - * all events either in the event_wait_list or all previously enqueued commands, queued + * Enqueues a barrier command which waits for either a list of events to complete, + * or if the list is empty it waits for all commands previously enqueued in command_queue + * to complete before it completes. This command blocks command execution, that is, any + * following commands enqueued after it do not execute until it completes. This command + * returns an event which can be waited on, i.e. this event can be waited on to insure that + * all events either in the event_wait_list or all previously enqueued commands, queued * before this command to command_queue, have completed. */ cl_int enqueueBarrierWithWaitList( @@ -7819,7 +7819,7 @@ public: return err; } - + /** * Enqueues a command to indicate with which device a set of memory objects * should be associated. @@ -7832,7 +7832,7 @@ public: ) { cl_event tmp; - + vector localMemObjects(memObjects.size()); for( int i = 0; i < (int)memObjects.size(); ++i ) { @@ -7842,8 +7842,8 @@ public: cl_int err = detail::errHandler( ::clEnqueueMigrateMemObjects( - object_, - (cl_uint)memObjects.size(), + object_, + (cl_uint)memObjects.size(), localMemObjects.data(), flags, (events != NULL) ? (cl_uint) events->size() : 0, @@ -7922,7 +7922,7 @@ public: for (unsigned int i = 0; i < elements; i++) { mems[i] = ((*mem_objects)[i])(); } - + cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueNativeKernel( @@ -7945,13 +7945,13 @@ public: * Deprecated APIs for 1.2 */ #if defined(CL_USE_DEPRECATED_OPENCL_1_1_APIS) - CL_EXT_PREFIX__VERSION_1_1_DEPRECATED + CL_EXT_PREFIX__VERSION_1_1_DEPRECATED cl_int enqueueMarker(Event* event = NULL) const CL_EXT_SUFFIX__VERSION_1_1_DEPRECATED { cl_event tmp; cl_int err = detail::errHandler( ::clEnqueueMarker( - object_, + object_, (event != NULL) ? &tmp : NULL), __ENQUEUE_MARKER_ERR); @@ -8042,7 +8042,7 @@ typedef CL_API_ENTRY cl_int (CL_API_CALL *PFN_clEnqueueReleaseD3D10ObjectsKHR)( #if CL_HPP_TARGET_OPENCL_VERSION >= 110 CL_HPP_INIT_CL_EXT_FCN_PTR_(clEnqueueAcquireD3D10ObjectsKHR); #endif - + cl_event tmp; cl_int err = detail::errHandler( pfn_clEnqueueAcquireD3D10ObjectsKHR( @@ -8210,7 +8210,7 @@ public: CL_QUEUE_OUT_OF_ORDER_EXEC_MODE_ENABLE | CL_QUEUE_ON_DEVICE | static_cast(properties); cl_queue_properties queue_properties[] = { CL_QUEUE_PROPERTIES, mergedProperties, - CL_QUEUE_SIZE, queueSize, + CL_QUEUE_SIZE, queueSize, 0 }; object_ = ::clCreateCommandQueueWithProperties( context(), device(), queue_properties, &error); @@ -8345,7 +8345,7 @@ public: } /*! - * Create a new default device command queue for the specified device + * Create a new default device command queue for the specified device * and of the requested size in bytes. * If there is already a default queue for the specified device this * function will return the pre-existing queue. @@ -8410,7 +8410,7 @@ Buffer::Buffer( if( useHostPtr ) { flags |= CL_MEM_USE_HOST_PTR; } - + size_type size = sizeof(DataType)*(endIterator - startIterator); if( useHostPtr ) { @@ -8583,7 +8583,7 @@ inline cl_int enqueueMapSVM( } /** - * Enqueues to the default queue a command that will allow the host to + * Enqueues to the default queue a command that will allow the host to * update a region of a coarse-grained SVM buffer. * This variant takes a cl::pointer instance. */ @@ -8661,7 +8661,7 @@ inline cl_int enqueueUnmapMemObject( #if CL_HPP_TARGET_OPENCL_VERSION >= 200 /** - * Enqueues to the default queue a command that will release a coarse-grained + * Enqueues to the default queue a command that will release a coarse-grained * SVM buffer back to the OpenCL runtime. * This variant takes a raw SVM pointer. */ @@ -8677,13 +8677,13 @@ inline cl_int enqueueUnmapSVM( return detail::errHandler(error, __ENQUEUE_UNMAP_MEM_OBJECT_ERR); } - return detail::errHandler(queue.enqueueUnmapSVM(ptr, events, event), + return detail::errHandler(queue.enqueueUnmapSVM(ptr, events, event), __ENQUEUE_UNMAP_MEM_OBJECT_ERR); } /** - * Enqueues to the default queue a command that will release a coarse-grained + * Enqueues to the default queue a command that will release a coarse-grained * SVM buffer back to the OpenCL runtime. * This variant takes a cl::pointer instance. */ @@ -8704,7 +8704,7 @@ inline cl_int enqueueUnmapSVM( } /** - * Enqueues to the default queue a command that will release a coarse-grained + * Enqueues to the default queue a command that will release a coarse-grained * SVM buffer back to the OpenCL runtime. * This variant takes a cl::vector instance. */ @@ -8787,11 +8787,11 @@ inline cl_int copy( const CommandQueue &queue, IteratorType startIterator, Itera { typedef typename std::iterator_traits::value_type DataType; cl_int error; - + size_type length = endIterator-startIterator; size_type byteLength = length*sizeof(DataType); - DataType *pointer = + DataType *pointer = static_cast(queue.enqueueMapBuffer(buffer, CL_TRUE, CL_MAP_WRITE, 0, byteLength, 0, 0, &error)); // if exceptions enabled, enqueueMapBuffer will throw if( error != CL_SUCCESS ) { @@ -8799,8 +8799,8 @@ inline cl_int copy( const CommandQueue &queue, IteratorType startIterator, Itera } #if defined(_MSC_VER) std::copy( - startIterator, - endIterator, + startIterator, + endIterator, stdext::checked_array_iterator( pointer, length)); #else @@ -8809,7 +8809,7 @@ inline cl_int copy( const CommandQueue &queue, IteratorType startIterator, Itera Event endEvent; error = queue.enqueueUnmapMemObject(buffer, pointer, 0, &endEvent); // if exceptions enabled, enqueueUnmapMemObject will throw - if( error != CL_SUCCESS ) { + if( error != CL_SUCCESS ) { return error; } endEvent.wait(); @@ -8826,11 +8826,11 @@ inline cl_int copy( const CommandQueue &queue, const cl::Buffer &buffer, Iterato { typedef typename std::iterator_traits::value_type DataType; cl_int error; - + size_type length = endIterator-startIterator; size_type byteLength = length*sizeof(DataType); - DataType *pointer = + DataType *pointer = static_cast(queue.enqueueMapBuffer(buffer, CL_TRUE, CL_MAP_READ, 0, byteLength, 0, 0, &error)); // if exceptions enabled, enqueueMapBuffer will throw if( error != CL_SUCCESS ) { @@ -8840,7 +8840,7 @@ inline cl_int copy( const CommandQueue &queue, const cl::Buffer &buffer, Iterato Event endEvent; error = queue.enqueueUnmapMemObject(buffer, pointer, 0, &endEvent); // if exceptions enabled, enqueueUnmapMemObject will throw - if( error != CL_SUCCESS ) { + if( error != CL_SUCCESS ) { return error; } endEvent.wait(); @@ -8892,17 +8892,17 @@ inline cl_int enqueueReadBufferRect( } return queue.enqueueReadBufferRect( - buffer, - blocking, - buffer_offset, + buffer, + blocking, + buffer_offset, host_offset, region, buffer_row_pitch, buffer_slice_pitch, host_row_pitch, host_slice_pitch, - ptr, - events, + ptr, + events, event); } @@ -8928,17 +8928,17 @@ inline cl_int enqueueWriteBufferRect( } return queue.enqueueWriteBufferRect( - buffer, - blocking, - buffer_offset, + buffer, + blocking, + buffer_offset, host_offset, region, buffer_row_pitch, buffer_slice_pitch, host_row_pitch, host_slice_pitch, - ptr, - events, + ptr, + events, event); } @@ -8972,7 +8972,7 @@ inline cl_int enqueueCopyBufferRect( src_slice_pitch, dst_row_pitch, dst_slice_pitch, - events, + events, event); } #endif // CL_HPP_TARGET_OPENCL_VERSION >= 110 @@ -8986,7 +8986,7 @@ inline cl_int enqueueReadImage( size_type slice_pitch, void* ptr, const vector* events = NULL, - Event* event = NULL) + Event* event = NULL) { cl_int error; CommandQueue queue = CommandQueue::getDefault(&error); @@ -9003,7 +9003,7 @@ inline cl_int enqueueReadImage( row_pitch, slice_pitch, ptr, - events, + events, event); } @@ -9033,7 +9033,7 @@ inline cl_int enqueueWriteImage( row_pitch, slice_pitch, ptr, - events, + events, event); } @@ -9135,7 +9135,7 @@ inline cl_int finish(void) if (error != CL_SUCCESS) { return error; - } + } return queue.finish(); @@ -9154,63 +9154,63 @@ private: friend class KernelFunctor; public: - EnqueueArgs(NDRange global) : + EnqueueArgs(NDRange global) : queue_(CommandQueue::getDefault()), - offset_(NullRange), + offset_(NullRange), global_(global), local_(NullRange) { } - EnqueueArgs(NDRange global, NDRange local) : + EnqueueArgs(NDRange global, NDRange local) : queue_(CommandQueue::getDefault()), - offset_(NullRange), + offset_(NullRange), global_(global), local_(local) { } - EnqueueArgs(NDRange offset, NDRange global, NDRange local) : + EnqueueArgs(NDRange offset, NDRange global, NDRange local) : queue_(CommandQueue::getDefault()), - offset_(offset), + offset_(offset), global_(global), local_(local) { } - EnqueueArgs(Event e, NDRange global) : + EnqueueArgs(Event e, NDRange global) : queue_(CommandQueue::getDefault()), - offset_(NullRange), + offset_(NullRange), global_(global), local_(NullRange) { events_.push_back(e); } - EnqueueArgs(Event e, NDRange global, NDRange local) : + EnqueueArgs(Event e, NDRange global, NDRange local) : queue_(CommandQueue::getDefault()), - offset_(NullRange), + offset_(NullRange), global_(global), local_(local) { events_.push_back(e); } - EnqueueArgs(Event e, NDRange offset, NDRange global, NDRange local) : + EnqueueArgs(Event e, NDRange offset, NDRange global, NDRange local) : queue_(CommandQueue::getDefault()), - offset_(offset), + offset_(offset), global_(global), local_(local) { events_.push_back(e); } - EnqueueArgs(const vector &events, NDRange global) : + EnqueueArgs(const vector &events, NDRange global) : queue_(CommandQueue::getDefault()), - offset_(NullRange), + offset_(NullRange), global_(global), local_(NullRange), events_(events) @@ -9218,9 +9218,9 @@ public: } - EnqueueArgs(const vector &events, NDRange global, NDRange local) : + EnqueueArgs(const vector &events, NDRange global, NDRange local) : queue_(CommandQueue::getDefault()), - offset_(NullRange), + offset_(NullRange), global_(global), local_(local), events_(events) @@ -9228,9 +9228,9 @@ public: } - EnqueueArgs(const vector &events, NDRange offset, NDRange global, NDRange local) : + EnqueueArgs(const vector &events, NDRange offset, NDRange global, NDRange local) : queue_(CommandQueue::getDefault()), - offset_(offset), + offset_(offset), global_(global), local_(local), events_(events) @@ -9238,63 +9238,63 @@ public: } - EnqueueArgs(CommandQueue &queue, NDRange global) : + EnqueueArgs(CommandQueue &queue, NDRange global) : queue_(queue), - offset_(NullRange), + offset_(NullRange), global_(global), local_(NullRange) { } - EnqueueArgs(CommandQueue &queue, NDRange global, NDRange local) : + EnqueueArgs(CommandQueue &queue, NDRange global, NDRange local) : queue_(queue), - offset_(NullRange), + offset_(NullRange), global_(global), local_(local) { } - EnqueueArgs(CommandQueue &queue, NDRange offset, NDRange global, NDRange local) : + EnqueueArgs(CommandQueue &queue, NDRange offset, NDRange global, NDRange local) : queue_(queue), - offset_(offset), + offset_(offset), global_(global), local_(local) { } - EnqueueArgs(CommandQueue &queue, Event e, NDRange global) : + EnqueueArgs(CommandQueue &queue, Event e, NDRange global) : queue_(queue), - offset_(NullRange), + offset_(NullRange), global_(global), local_(NullRange) { events_.push_back(e); } - EnqueueArgs(CommandQueue &queue, Event e, NDRange global, NDRange local) : + EnqueueArgs(CommandQueue &queue, Event e, NDRange global, NDRange local) : queue_(queue), - offset_(NullRange), + offset_(NullRange), global_(global), local_(local) { events_.push_back(e); } - EnqueueArgs(CommandQueue &queue, Event e, NDRange offset, NDRange global, NDRange local) : + EnqueueArgs(CommandQueue &queue, Event e, NDRange offset, NDRange global, NDRange local) : queue_(queue), - offset_(offset), + offset_(offset), global_(global), local_(local) { events_.push_back(e); } - EnqueueArgs(CommandQueue &queue, const vector &events, NDRange global) : + EnqueueArgs(CommandQueue &queue, const vector &events, NDRange global) : queue_(queue), - offset_(NullRange), + offset_(NullRange), global_(global), local_(NullRange), events_(events) @@ -9302,9 +9302,9 @@ public: } - EnqueueArgs(CommandQueue &queue, const vector &events, NDRange global, NDRange local) : + EnqueueArgs(CommandQueue &queue, const vector &events, NDRange global, NDRange local) : queue_(queue), - offset_(NullRange), + offset_(NullRange), global_(global), local_(local), events_(events) @@ -9312,9 +9312,9 @@ public: } - EnqueueArgs(CommandQueue &queue, const vector &events, NDRange offset, NDRange global, NDRange local) : + EnqueueArgs(CommandQueue &queue, const vector &events, NDRange offset, NDRange global, NDRange local) : queue_(queue), - offset_(offset), + offset_(offset), global_(global), local_(local), events_(events) @@ -9329,7 +9329,7 @@ public: /** * Type safe kernel functor. - * + * */ template class KernelFunctor @@ -9381,7 +9381,7 @@ public: { Event event; setArgs<0>(std::forward(ts)...); - + args.queue_.enqueueNDRangeKernel( kernel_, args.offset_, @@ -9414,7 +9414,7 @@ public: args.local_, &args.events_, &event); - + return event; } diff --git a/include/openpose/gpu/cuda.hu b/include/openpose_private/gpu/cuda.hu similarity index 98% rename from include/openpose/gpu/cuda.hu rename to include/openpose_private/gpu/cuda.hu index 1575f590..02512303 100644 --- a/include/openpose/gpu/cuda.hu +++ b/include/openpose_private/gpu/cuda.hu @@ -1,5 +1,5 @@ -#ifndef OPENPOSE_GPU_CUDA_HU -#define OPENPOSE_GPU_CUDA_HU +#ifndef OPENPOSE_PRIVATE_GPU_CUDA_HU +#define OPENPOSE_PRIVATE_GPU_CUDA_HU // Note: This class should only be included if CUDA is enabled @@ -201,4 +201,4 @@ namespace op } } -#endif // OPENPOSE_GPU_CUDA_HU +#endif // OPENPOSE_PRIVATE_GPU_CUDA_HU diff --git a/include/openpose/gpu/opencl.hcl b/include/openpose_private/gpu/opencl.hcl similarity index 92% rename from include/openpose/gpu/opencl.hcl rename to include/openpose_private/gpu/opencl.hcl index a8579af2..82414d1d 100644 --- a/include/openpose/gpu/opencl.hcl +++ b/include/openpose_private/gpu/opencl.hcl @@ -1,5 +1,5 @@ -#ifndef OPENPOSE_CORE_OPENCL_HPP -#define OPENPOSE_CORE_OPENCL_HPP +#ifndef OPENPOSE_PRIVATE_GPU_OPENCL_HPP +#define OPENPOSE_PRIVATE_GPU_OPENCL_HPP #include @@ -30,7 +30,7 @@ namespace cl namespace op { - class OP_API OpenCL + class OpenCL { public: static std::shared_ptr getInstance(const int deviceId = 0, const int deviceType = CL_DEVICE_TYPE_GPU, @@ -51,10 +51,10 @@ namespace op template inline K getKernelFunctorFromManager(const std::string& kernelName, const std::string& src = "", bool isFile = false) - { + { return K(getKernelFromManager(kernelName, src, isFile)); } - + template static void getBufferRegion(cl_buffer_region& region, const int origin, const int size); int getAlignment(); @@ -73,4 +73,4 @@ namespace op }; } -#endif // OPENPOSE_CORE_OPENCL_HPP +#endif // OPENPOSE_PRIVATE_GPU_OPENCL_HPP diff --git a/include/openpose_private/tracking/pyramidalLK.hpp b/include/openpose_private/tracking/pyramidalLK.hpp new file mode 100644 index 00000000..d32c307d --- /dev/null +++ b/include/openpose_private/tracking/pyramidalLK.hpp @@ -0,0 +1,27 @@ +#ifndef OPENPOSE_PRIVATE_TRACKING_LKPYRAMIDAL_HPP +#define OPENPOSE_PRIVATE_TRACKING_LKPYRAMIDAL_HPP + +#include // cv::Mat, cv::Point2f +#include + +namespace op +{ + void pyramidalLKCpu( + std::vector& coordI, std::vector& coordJ, + std::vector& pyramidImagesPrevious, std::vector& pyramidImagesCurrent, + std::vector& status, const cv::Mat& imagePrevious, const cv::Mat& imageCurrent, + const int levels = 3, const int patchSize = 21); + + int pyramidalLKGpu( + std::vector& ptsI, std::vector& ptsJ, + std::vector& status, const cv::Mat& imagePrevious, const cv::Mat& imageCurrent, + const int levels = 3, const int patchSize = 21); + + void pyramidalLKOcv( + std::vector& coordI, std::vector& coordJ, + std::vector& pyramidImagesPrevious, std::vector& pyramidImagesCurrent, + std::vector& status, const cv::Mat& imagePrevious, const cv::Mat& imageCurrent, + const int levels = 3, const int patchSize = 21, const bool initFlow = false); +} + +#endif // OPENPOSE_PRIVATE_TRACKING_LKPYRAMIDAL_HPP diff --git a/include/openpose/utilities/avx.hpp b/include/openpose_private/utilities/avx.hpp similarity index 96% rename from include/openpose/utilities/avx.hpp rename to include/openpose_private/utilities/avx.hpp index 40638f78..cca24d35 100644 --- a/include/openpose/utilities/avx.hpp +++ b/include/openpose_private/utilities/avx.hpp @@ -1,5 +1,5 @@ -#ifndef OPENPOSE_UTILITIES_AVX_HPP -#define OPENPOSE_UTILITIES_AVX_HPP +#ifndef OPENPOSE_PRIVATE_UTILITIES_AVX_HPP +#define OPENPOSE_PRIVATE_UTILITIES_AVX_HPP // Warning: // This file contains auxiliary functions for AVX. @@ -97,4 +97,4 @@ } #endif -#endif // OPENPOSE_UTILITIES_AVX_HPP +#endif // OPENPOSE_PRIVATE_UTILITIES_AVX_HPP diff --git a/include/openpose_private/utilities/openCvMultiversionHeaders.hpp b/include/openpose_private/utilities/openCvMultiversionHeaders.hpp new file mode 100644 index 00000000..9f193f64 --- /dev/null +++ b/include/openpose_private/utilities/openCvMultiversionHeaders.hpp @@ -0,0 +1,61 @@ +#ifndef OPENPOSE_PRIVATE_UTILITIES_OPENCV_MULTIVERSION_HEADERS_HPP +#define OPENPOSE_PRIVATE_UTILITIES_OPENCV_MULTIVERSION_HEADERS_HPP + +#include + +// Compabitility for OpenCV 4.0 while preserving 2.4.X and 3.X HEADERS +// Note: +// - CV_VERSION: 2.4.9.1 | 4.0.0-beta +// - CV_MAJOR_VERSION: 2 | 4 +// - CV_MINOR_VERSION: 4 | 0 +// - CV_SUBMINOR_VERSION: 9 | 0 +// - CV_VERSION_EPOCH: 2 | Not defined +#if (defined(CV_MAJOR_VERSION) && CV_MAJOR_VERSION > 3) + #define OPEN_CV_IS_4_OR_HIGHER +#endif +#ifdef OPEN_CV_IS_4_OR_HIGHER + #define CV_BGR2GRAY cv::COLOR_BGR2GRAY + #define CV_BGR2RGB cv::COLOR_BGR2RGB + #define CV_CALIB_CB_ADAPTIVE_THRESH cv::CALIB_CB_ADAPTIVE_THRESH + #define CV_CALIB_CB_NORMALIZE_IMAGE cv::CALIB_CB_NORMALIZE_IMAGE + #define CV_CALIB_CB_FILTER_QUADS cv::CALIB_CB_FILTER_QUADS + #define CV_CAP_PROP_FPS cv::CAP_PROP_FPS + #define CV_CAP_PROP_FRAME_COUNT cv::CAP_PROP_FRAME_COUNT + #define CV_CAP_PROP_FRAME_HEIGHT cv::CAP_PROP_FRAME_HEIGHT + #define CV_CAP_PROP_FRAME_WIDTH cv::CAP_PROP_FRAME_WIDTH + #define CV_CAP_PROP_POS_FRAMES cv::CAP_PROP_POS_FRAMES + #define CV_FOURCC cv::VideoWriter::fourcc + #define CV_GRAY2BGR cv::COLOR_GRAY2BGR + #define CV_HAAR_SCALE_IMAGE cv::CASCADE_SCALE_IMAGE + #define CV_INTER_CUBIC cv::INTER_CUBIC + #define CV_INTER_LINEAR cv::INTER_LINEAR + #define CV_L2 cv::NORM_L2 + #define CV_RGB2BGR cv::COLOR_RGB2BGR + #define CV_TERMCRIT_EPS cv::TermCriteria::Type::EPS + #define CV_TERMCRIT_ITER cv::TermCriteria::Type::MAX_ITER + #define CV_WARP_INVERSE_MAP cv::WARP_INVERSE_MAP + #define CV_WINDOW_FULLSCREEN cv::WINDOW_FULLSCREEN + #define CV_WINDOW_KEEPRATIO cv::WINDOW_KEEPRATIO + #define CV_WINDOW_NORMAL cv::WINDOW_NORMAL + #define CV_WINDOW_OPENGL cv::WINDOW_OPENGL + #define CV_WND_PROP_FULLSCREEN cv::WND_PROP_FULLSCREEN + // Required for alpha and beta versions, but not for rc version + #include + #ifndef CV_IMWRITE_JPEG_QUALITY + #define CV_IMWRITE_JPEG_QUALITY cv::IMWRITE_JPEG_QUALITY + #endif + #ifndef CV_IMWRITE_PNG_COMPRESSION + #define CV_IMWRITE_PNG_COMPRESSION cv::IMWRITE_PNG_COMPRESSION + #endif + #ifndef CV_LOAD_IMAGE_ANYDEPTH + #define CV_LOAD_IMAGE_ANYDEPTH cv::IMREAD_ANYDEPTH + #endif + #ifndef CV_LOAD_IMAGE_COLOR + #define CV_LOAD_IMAGE_COLOR cv::IMREAD_COLOR + #endif + #ifndef CV_LOAD_IMAGE_GRAYSCALE + #define CV_LOAD_IMAGE_GRAYSCALE cv::IMREAD_GRAYSCALE + #endif +#endif + +#endif // OPENPOSE_PRIVATE_UTILITIES_OPENCV_MULTIVERSION_HEADERS_HPP diff --git a/include/openpose_private/utilities/openCvPrivate.hpp b/include/openpose_private/utilities/openCvPrivate.hpp new file mode 100644 index 00000000..70501589 --- /dev/null +++ b/include/openpose_private/utilities/openCvPrivate.hpp @@ -0,0 +1,19 @@ +#ifndef OPENPOSE_PRIVATE_UTILITIES_OPEN_CV_PRIVATE_HPP +#define OPENPOSE_PRIVATE_UTILITIES_OPEN_CV_PRIVATE_HPP + +#include // cv::Mat, cv::Rect, cv::Scalar +#include // cv::BORDER_CONSTANT +#include + +namespace op +{ + void putTextOnCvMat( + cv::Mat& cvMat, const std::string& textToDisplay, const Point& position, + const cv::Scalar& color, const bool normalizeWidth, const int imageWidth); + + void resizeFixedAspectRatio( + cv::Mat& resizedCvMat, const cv::Mat& cvMat, const double scaleFactor, const Point& targetSize, + const int borderMode = cv::BORDER_CONSTANT, const cv::Scalar& borderValue = cv::Scalar{0,0,0}); +} + +#endif // OPENPOSE_PRIVATE_UTILITIES_OPEN_CV_PRIVATE_HPP diff --git a/include/openpose/utilities/render.hu b/include/openpose_private/utilities/render.hu similarity index 99% rename from include/openpose/utilities/render.hu rename to include/openpose_private/utilities/render.hu index 0ff4791f..ad743c67 100644 --- a/include/openpose/utilities/render.hu +++ b/include/openpose_private/utilities/render.hu @@ -1,5 +1,5 @@ -#ifndef OPENPOSE_UTILITIES_RENDER_HU -#define OPENPOSE_UTILITIES_RENDER_HU +#ifndef OPENPOSE_PRIVATE_UTILITIES_RENDER_HU +#define OPENPOSE_PRIVATE_UTILITIES_RENDER_HU namespace op { @@ -377,4 +377,4 @@ namespace op } -#endif // OPENPOSE_UTILITIES_RENDER_HU +#endif // OPENPOSE_PRIVATE_UTILITIES_RENDER_HU diff --git a/src/openpose/3d/CMakeLists.txt b/src/openpose/3d/CMakeLists.txt index f06fdb2d..df2d7cc3 100644 --- a/src/openpose/3d/CMakeLists.txt +++ b/src/openpose/3d/CMakeLists.txt @@ -2,7 +2,8 @@ set(SOURCES_OP_3D cameraParameterReader.cpp defineTemplates.cpp jointAngleEstimation.cpp - poseTriangulation.cpp) + poseTriangulation.cpp + poseTriangulationPrivate.cpp) include(${CMAKE_SOURCE_DIR}/cmake/Utils.cmake) prepend(SOURCES_OP_3D_WITH_CP ${CMAKE_CURRENT_SOURCE_DIR} ${SOURCES_OP_3D}) @@ -17,7 +18,7 @@ if (UNIX OR APPLE) endif () add_library(caffe SHARED IMPORTED) - set_property(TARGET caffe PROPERTY IMPORTED_LOCATION ${Caffe_LIBS}) + set_property(TARGET caffe PROPERTY IMPORTED_LOCATION ${Caffe_LIBS}) target_link_libraries(openpose_3d caffe openpose_core ${MKL_LIBS}) if (BUILD_CAFFE) diff --git a/src/openpose/3d/cameraParameterReader.cpp b/src/openpose/3d/cameraParameterReader.cpp index e6b6b626..183c97fc 100644 --- a/src/openpose/3d/cameraParameterReader.cpp +++ b/src/openpose/3d/cameraParameterReader.cpp @@ -1,16 +1,36 @@ -#include // OPEN_CV_IS_4_OR_HIGHER +#include +#include // OPEN_CV_IS_4_OR_HIGHER #ifdef OPEN_CV_IS_4_OR_HIGHER #include // cv::initUndistortRectifyMap in OpenCV 4 #endif #include // cv::initUndistortRectifyMap (OpenCV <= 3), cv::undistort #include #include -#include namespace op { + struct CameraParameterReader::ImplCameraParameterReader + { + std::vector mSerialNumbers; + std::vector mCameraMatrices; + std::vector mCameraDistortions; + std::vector mCameraIntrinsics; + std::vector mCameraExtrinsics; + std::vector mCameraExtrinsicsInitial; + + // Undistortion (optional) + bool mUndistortImage; + std::vector mRemoveDistortionMaps1; + std::vector mRemoveDistortionMaps2; + + ImplCameraParameterReader(const bool undistortImage) : + mUndistortImage{undistortImage} + { + } + }; + CameraParameterReader::CameraParameterReader() : - mUndistortImage{false} + spImpl{std::make_shared(false)} { } @@ -19,11 +39,11 @@ namespace op } CameraParameterReader::CameraParameterReader(const std::string& serialNumber, - const cv::Mat& cameraIntrinsics, - const cv::Mat& cameraDistortion, - const cv::Mat& cameraExtrinsics, - const cv::Mat& cameraExtrinsicsInitial) : - mUndistortImage{false} + const Matrix& cameraIntrinsics, + const Matrix& cameraDistortion, + const Matrix& cameraExtrinsics, + const Matrix& cameraExtrinsicsInitial) : + spImpl{std::make_shared(false)} { try { @@ -35,24 +55,26 @@ namespace op if (cameraDistortion.empty()) error("Camera distortion cannot be empty.", __LINE__, __FUNCTION__, __FILE__); // Add new matrices - mSerialNumbers.emplace_back(serialNumber); - mCameraIntrinsics.emplace_back(cameraIntrinsics.clone()); - mCameraDistortions.emplace_back(cameraDistortion.clone()); + spImpl->mSerialNumbers.emplace_back(serialNumber); + spImpl->mCameraIntrinsics.emplace_back(cameraIntrinsics.clone()); + spImpl->mCameraDistortions.emplace_back(cameraDistortion.clone()); // Add extrinsics if not empty if (!cameraExtrinsics.empty()) - mCameraExtrinsics.emplace_back(cameraExtrinsics.clone()); + spImpl->mCameraExtrinsics.emplace_back(cameraExtrinsics.clone()); else - mCameraExtrinsics.emplace_back(cv::Mat::eye(3, 4, cameraIntrinsics.type())); + spImpl->mCameraExtrinsics.emplace_back(Matrix::eye(3, 4, cameraIntrinsics.type())); // Add extrinsics (initial) if not empty if (!cameraExtrinsicsInitial.empty()) - mCameraExtrinsicsInitial.emplace_back(cameraExtrinsicsInitial.clone()); + spImpl->mCameraExtrinsicsInitial.emplace_back(cameraExtrinsicsInitial.clone()); // Otherwise, add cv::eye else - mCameraExtrinsicsInitial.emplace_back(cv::Mat::eye(3, 4, cameraIntrinsics.type())); - mCameraMatrices.emplace_back(mCameraIntrinsics.back() * mCameraExtrinsics.back()); - // Undistortion cv::Mats - mRemoveDistortionMaps1.resize(getNumberCameras()); - mRemoveDistortionMaps2.resize(getNumberCameras()); + spImpl->mCameraExtrinsicsInitial.emplace_back(Matrix::eye(3, 4, cameraIntrinsics.type()));; + const cv::Mat cvCameraMatrices = OP_OP2CVCONSTMAT(spImpl->mCameraIntrinsics.back()) * OP_OP2CVCONSTMAT(spImpl->mCameraExtrinsics.back()); + const Matrix opCameraMatrices = OP_CV2OPCONSTMAT(cvCameraMatrices); + spImpl->mCameraMatrices.emplace_back(opCameraMatrices); + // Undistortion Mats + spImpl->mRemoveDistortionMaps1.resize(getNumberCameras()); + spImpl->mRemoveDistortionMaps2.resize(getNumberCameras()); } catch (const std::exception& e) { @@ -68,12 +90,12 @@ namespace op // Serial numbers if (serialNumbers.empty()) { - mSerialNumbers = getFilesOnDirectory(cameraParameterPath, "xml"); - for (auto& serialNumber : mSerialNumbers) + spImpl->mSerialNumbers = getFilesOnDirectory(cameraParameterPath, "xml"); + for (auto& serialNumber : spImpl->mSerialNumbers) serialNumber = getFileNameNoExtension(serialNumber); } else - mSerialNumbers = serialNumbers; + spImpl->mSerialNumbers = serialNumbers; // Commong saving/loading const auto dataFormat = DataFormat::Xml; @@ -82,22 +104,23 @@ namespace op }; // Load parameters - mCameraMatrices.clear(); - mCameraDistortions.clear(); - mCameraIntrinsics.clear(); - mCameraExtrinsics.clear(); - mCameraExtrinsicsInitial.clear(); + spImpl->mCameraMatrices.clear(); + spImpl->mCameraDistortions.clear(); + spImpl->mCameraIntrinsics.clear(); + spImpl->mCameraExtrinsics.clear(); + spImpl->mCameraExtrinsicsInitial.clear(); // log("Camera matrices:"); - for (auto i = 0ull ; i < mSerialNumbers.size() ; i++) + for (auto i = 0ull ; i < spImpl->mSerialNumbers.size() ; i++) { - const auto parameterPath = cameraParameterPath + mSerialNumbers.at(i); - const auto cameraParameters = loadData(cvMatNames, parameterPath, dataFormat); + const auto parameterPath = cameraParameterPath + spImpl->mSerialNumbers.at(i); + const auto opCameraParameters = loadData(cvMatNames, parameterPath, dataFormat); + OP_OP2CVVECTORMAT(cameraParameters, opCameraParameters) // Error if empty element if (cameraParameters.empty() || cameraParameters.at(0).empty() || cameraParameters.at(1).empty() || cameraParameters.at(2).empty() || cameraParameters.at(3).empty()) { - const std::string errorMessage = " of the camera with serial number `" + mSerialNumbers[i] + const std::string errorMessage = " of the camera with serial number `" + spImpl->mSerialNumbers[i] + "` (file: " + parameterPath + "." + dataFormatToString(dataFormat) + "). Is its format valid? You might want to check the example xml" + " file."; @@ -118,27 +141,29 @@ namespace op // error("Error at reading the camera distortion parameters" + errorMessage, // __LINE__, __FUNCTION__, __FILE__); } - mCameraExtrinsics.emplace_back(cameraParameters.at(0)); - mCameraIntrinsics.emplace_back(cameraParameters.at(1)); - mCameraDistortions.emplace_back(cameraParameters.at(2)); - mCameraExtrinsicsInitial.emplace_back(cameraParameters.at(3)); - mCameraMatrices.emplace_back(mCameraIntrinsics.back() * mCameraExtrinsics.back()); + spImpl->mCameraExtrinsics.emplace_back(opCameraParameters.at(0)); + spImpl->mCameraIntrinsics.emplace_back(opCameraParameters.at(1)); + spImpl->mCameraDistortions.emplace_back(opCameraParameters.at(2)); + spImpl->mCameraExtrinsicsInitial.emplace_back(opCameraParameters.at(3)); + const cv::Mat cvCameraMatrices = OP_OP2CVCONSTMAT(spImpl->mCameraIntrinsics.back()) * OP_OP2CVCONSTMAT(spImpl->mCameraExtrinsics.back()); + const Matrix opCameraMatrices = OP_CV2OPCONSTMAT(cvCameraMatrices); + spImpl->mCameraMatrices.emplace_back(opCameraMatrices); // log(cameraParameters.at(0)); } - // Undistortion cv::Mats - mRemoveDistortionMaps1.resize(getNumberCameras()); - mRemoveDistortionMaps2.resize(getNumberCameras()); - // // mCameraMatrices + // Undistortion Mats + spImpl->mRemoveDistortionMaps1.resize(getNumberCameras()); + spImpl->mRemoveDistortionMaps2.resize(getNumberCameras()); + // // spImpl->mCameraMatrices // log("\nFull camera matrices:"); - // for (const auto& cvMat : mCameraMatrices) + // for (const auto& cvMat : spImpl->mCameraMatrices) // log(cvMat); - // // mCameraIntrinsics + // // spImpl->mCameraIntrinsics // log("\nCamera intrinsic parameters:"); - // for (const auto& cvMat : mCameraIntrinsics) + // for (const auto& cvMat : spImpl->mCameraIntrinsics) // log(cvMat); - // // mCameraDistortions + // // spImpl->mCameraDistortions // log("\nCamera distortion parameters:"); - // for (const auto& cvMat : mCameraDistortions) + // for (const auto& cvMat : spImpl->mCameraDistortions) // log(cvMat); } catch (const std::exception& e) @@ -165,24 +190,26 @@ namespace op try { // Sanity check - if (mSerialNumbers.size() != mCameraIntrinsics.size() || mSerialNumbers.size() != mCameraDistortions.size() - || (mSerialNumbers.size() != mCameraIntrinsics.size() && !mCameraExtrinsics.empty())) - error("Arguments must have same size (mSerialNumbers, mCameraIntrinsics, mCameraDistortions," - " and mCameraExtrinsics).", __LINE__, __FUNCTION__, __FILE__); + if (spImpl->mSerialNumbers.size() != spImpl->mCameraIntrinsics.size() + || spImpl->mSerialNumbers.size() != spImpl->mCameraDistortions.size() + || (spImpl->mSerialNumbers.size() != spImpl->mCameraIntrinsics.size() + && !spImpl->mCameraExtrinsics.empty())) + error("Arguments must have same size (spImpl->mSerialNumbers, spImpl->mCameraIntrinsics, spImpl->mCameraDistortions," + " and spImpl->mCameraExtrinsics).", __LINE__, __FUNCTION__, __FILE__); // Commong saving/loading const auto dataFormat = DataFormat::Xml; const std::vector cvMatNames { "CameraMatrix", "Intrinsics", "Distortion", "CameraMatrixInitial" }; // Saving - for (auto i = 0ull ; i < mSerialNumbers.size() ; i++) + for (auto i = 0ull ; i < spImpl->mSerialNumbers.size() ; i++) { - std::vector cameraParameters; - cameraParameters.emplace_back(mCameraExtrinsics[i]); - cameraParameters.emplace_back(mCameraIntrinsics[i]); - cameraParameters.emplace_back(mCameraDistortions[i]); - cameraParameters.emplace_back(mCameraExtrinsicsInitial[i]); - saveData(cameraParameters, cvMatNames, cameraParameterPath + mSerialNumbers[i], dataFormat); + std::vector cameraParameters; + cameraParameters.emplace_back(spImpl->mCameraExtrinsics[i]); + cameraParameters.emplace_back(spImpl->mCameraIntrinsics[i]); + cameraParameters.emplace_back(spImpl->mCameraDistortions[i]); + cameraParameters.emplace_back(spImpl->mCameraExtrinsicsInitial[i]); + saveData(cameraParameters, cvMatNames, cameraParameterPath + spImpl->mSerialNumbers[i], dataFormat); } } catch (const std::exception& e) @@ -195,7 +222,7 @@ namespace op { try { - return mSerialNumbers.size(); + return spImpl->mSerialNumbers.size(); } catch (const std::exception& e) { @@ -208,77 +235,77 @@ namespace op { try { - return mSerialNumbers; + return spImpl->mSerialNumbers; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return mSerialNumbers; + return spImpl->mSerialNumbers; } } - const std::vector& CameraParameterReader::getCameraMatrices() const + const std::vector& CameraParameterReader::getCameraMatrices() const { try { - return mCameraMatrices; + return spImpl->mCameraMatrices; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return mCameraMatrices; + return spImpl->mCameraMatrices; } } - const std::vector& CameraParameterReader::getCameraDistortions() const + const std::vector& CameraParameterReader::getCameraDistortions() const { try { - return mCameraDistortions; + return spImpl->mCameraDistortions; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return mCameraDistortions; + return spImpl->mCameraDistortions; } } - const std::vector& CameraParameterReader::getCameraIntrinsics() const + const std::vector& CameraParameterReader::getCameraIntrinsics() const { try { - return mCameraIntrinsics; + return spImpl->mCameraIntrinsics; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return mCameraIntrinsics; + return spImpl->mCameraIntrinsics; } } - const std::vector& CameraParameterReader::getCameraExtrinsics() const + const std::vector& CameraParameterReader::getCameraExtrinsics() const { try { - return mCameraExtrinsics; + return spImpl->mCameraExtrinsics; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return mCameraExtrinsics; + return spImpl->mCameraExtrinsics; } } - const std::vector& CameraParameterReader::getCameraExtrinsicsInitial() const + const std::vector& CameraParameterReader::getCameraExtrinsicsInitial() const { try { - return mCameraExtrinsicsInitial; + return spImpl->mCameraExtrinsicsInitial; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return mCameraExtrinsicsInitial; + return spImpl->mCameraExtrinsicsInitial; } } @@ -286,7 +313,7 @@ namespace op { try { - return mUndistortImage; + return spImpl->mUndistortImage; } catch (const std::exception& e) { @@ -299,7 +326,7 @@ namespace op { try { - mUndistortImage = undistortImage; + spImpl->mUndistortImage = undistortImage; } catch (const std::exception& e) { @@ -307,56 +334,62 @@ namespace op } } - void CameraParameterReader::undistort(cv::Mat& frame, const unsigned int cameraIndex) + void CameraParameterReader::undistort(Matrix& frame, const unsigned int cameraIndex) { try { - if (mUndistortImage) + if (spImpl->mUndistortImage) { // Sanity check - if (mRemoveDistortionMaps1.size() <= cameraIndex || mRemoveDistortionMaps2.size() <= cameraIndex) + if (spImpl->mRemoveDistortionMaps1.size() <= cameraIndex + || spImpl->mRemoveDistortionMaps2.size() <= cameraIndex) { - error("Variable cameraIndex is out of bounds, it should be smaller than mRemoveDistortionMapsX.", + error("Variable cameraIndex is out of bounds, it should be smaller than spImpl->mRemoveDistortionMapsX.", __LINE__, __FUNCTION__, __FILE__); } // Only first time - if (mRemoveDistortionMaps1[cameraIndex].empty() || mRemoveDistortionMaps2[cameraIndex].empty()) + if (spImpl->mRemoveDistortionMaps1[cameraIndex].empty() + || spImpl->mRemoveDistortionMaps2[cameraIndex].empty()) { - const auto cameraIntrinsics = mCameraIntrinsics.at(0); - const auto cameraDistorsions = mCameraDistortions.at(0); - const auto imageSize = frame.size(); + const auto cvCameraIntrinsics = OP_OP2CVCONSTMAT(spImpl->mCameraIntrinsics.at(0)); + const auto cvCameraDistorsions = OP_OP2CVCONSTMAT(spImpl->mCameraDistortions.at(0)); + //const auto imageSize = OP_OP2CVMAT(frame).size(); + cv::Size imageSize; + OP_CONST_MAT_RETURN_FUNCTION(imageSize, frame, size()); // = frame.size(); // // Option a - 80 ms / 3 images // // http://docs.opencv.org/2.4/modules/imgproc/doc/geometric_transformations.html#undistort - // cv::undistort(cvMatDistorted, mCvMats[i], cameraIntrinsics, cameraDistorsions); + // cv::undistort(cvMatDistorted, mCvMats[i], cvCameraIntrinsics, cvCameraDistorsions); // // In OpenCV 2.4, cv::undistort is exactly equal than cv::initUndistortRectifyMap // (with CV_16SC2) + cv::remap (with LINEAR). I.e., log(cv::norm(cvMatMethod1-cvMatMethod2)) = 0. // Option b - 15 ms / 3 images (LINEAR) or 25 ms (CUBIC) // Distorsion removal - not required and more expensive (applied to the whole image instead of // only to our interest points) cv::initUndistortRectifyMap( - cameraIntrinsics, cameraDistorsions, cv::Mat(), - // cameraIntrinsics instead of cv::getOptimalNewCameraMatrix to + cvCameraIntrinsics, cvCameraDistorsions, cv::Mat(), + // cvCameraIntrinsics instead of cv::getOptimalNewCameraMatrix to // avoid black borders - cameraIntrinsics, + cvCameraIntrinsics, // #include for next line - // cv::getOptimalNewCameraMatrix(cameraIntrinsics, - // cameraDistorsions, + // cv::getOptimalNewCameraMatrix(cvCameraIntrinsics, + // cvCameraDistorsions, // imageSize, 1, // imageSize, 0), imageSize, CV_16SC2, // Faster, less memory // CV_32FC1, // More accurate - mRemoveDistortionMaps1[cameraIndex], - mRemoveDistortionMaps2[cameraIndex]); + spImpl->mRemoveDistortionMaps1[cameraIndex], + spImpl->mRemoveDistortionMaps2[cameraIndex]); } cv::Mat undistortedCvMat; - cv::remap(frame, undistortedCvMat, - mRemoveDistortionMaps1[cameraIndex], mRemoveDistortionMaps2[cameraIndex], + const cv::Mat cvFrame = OP_OP2CVCONSTMAT(frame); + cv::remap(cvFrame, undistortedCvMat, + spImpl->mRemoveDistortionMaps1[cameraIndex], spImpl->mRemoveDistortionMaps2[cameraIndex], // cv::INTER_NEAREST); cv::INTER_LINEAR); // cv::INTER_CUBIC); // cv::INTER_LANCZOS4); // Smoother, but we do not need this quality & its >>expensive - std::swap(undistortedCvMat, frame); + Matrix opUndistortedCvMat = OP_CV2OPMAT(undistortedCvMat); + std::swap(opUndistortedCvMat, frame); } } catch (const std::exception& e) diff --git a/src/openpose/3d/poseTriangulation.cpp b/src/openpose/3d/poseTriangulation.cpp index 83a9084a..2de3ae19 100644 --- a/src/openpose/3d/poseTriangulation.cpp +++ b/src/openpose/3d/poseTriangulation.cpp @@ -1,321 +1,10 @@ -#include // std::accumulate -#ifdef USE_CERES - #include - #include -#endif -#include -#include #include +#include // std::accumulate +#include +#include namespace op { - double calcReprojectionError(const cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, - const std::vector& pointsOnEachCamera) - { - try - { - auto averageError = 0.; - for (auto i = 0u ; i < cameraMatrices.size() ; i++) - { - cv::Mat imageX = cameraMatrices[i] * reconstructedPoint; - imageX /= imageX.at(2,0); - const auto error = std::sqrt(std::pow(imageX.at(0,0) - pointsOnEachCamera[i].x,2) - + std::pow(imageX.at(1,0) - pointsOnEachCamera[i].y,2)); - // log("Error: " + std::to_string(error)); - averageError += error; - } - return averageError / cameraMatrices.size(); - } - catch (const std::exception& e) - { - error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return -1.; - } - } - - #ifdef USE_CERES - // Nonlinear Optimization for 3D Triangulation - struct ReprojectionErrorForTriangulation - { - ReprojectionErrorForTriangulation(const double x, const double y, const double* const param) : - observed_x{x}, - observed_y{y} - { - memcpy(camParam, param, sizeof(double)*12); - } - - template - bool operator()(const T* const pt, - T* residuals) const ; - - inline virtual bool Evaluate(double const* const* pt, - double* residuals, - double** jacobians) const; - - const double observed_x; - const double observed_y; - double camParam[12]; - }; - - template - bool ReprojectionErrorForTriangulation::operator()(const T* const pt, - T* residuals) const - { - try - { - const T predicted[3] = { - T(camParam[0])*pt[0] + T(camParam[1])*pt[1] + T(camParam[2])*pt[2] + T(camParam[3]), - T(camParam[4])*pt[0] + T(camParam[5])*pt[1] + T(camParam[6])*pt[2] + T(camParam[7]), - T(camParam[8])*pt[0] + T(camParam[9])*pt[1] + T(camParam[10])*pt[2] + T(camParam[11])}; - - residuals[0] = T(observed_x) - predicted[0] / predicted[2]; - residuals[1] = T(observed_y) - predicted[1] / predicted[2]; - - // residuals[0] = T(pow(predicted[0] - observed_x,2) + pow(predicted[1] - observed_y,2)); - // residuals[0] = -pow(predicted[0] - T(observed_x),2); - // residuals[1] = -pow(predicted[1] - T(observed_y),2); - - return true; - } - catch (const std::exception& e) - { - error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return false; - } - } - - bool ReprojectionErrorForTriangulation::Evaluate(double const* const* pt, - double* residuals, - double** jacobians) const - { - try - { - UNUSED(jacobians); - - const double predicted[3] = { - camParam[0]*pt[0][0] + camParam[1]*pt[0][1] + camParam[2]*pt[0][2] + camParam[3], - camParam[4]*pt[0][0] + camParam[5]*pt[0][1] + camParam[6]*pt[0][2] + camParam[7], - camParam[8]*pt[0][0] + camParam[9]*pt[0][1] + camParam[10]*pt[0][2] + camParam[11]}; - - // residuals[0] = predicted[0] / predicted[2] - observed_x; - // residuals[1] = predicted[1] / predicted[2] - observed_y; - - residuals[0] = std::sqrt(std::pow(predicted[0] / predicted[2] - observed_x,2) - + std::pow(predicted[1] / predicted[2] - observed_y,2)); - - // log("Residuals:"); - // residuals[0]= pow(predicted[0] - (observed_x),2); - // residuals[1]= pow(predicted[1] - (observed_y),2); - - return true; - } - catch (const std::exception& e) - { - error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return false; - } - } - #endif - - double triangulate(cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, - const std::vector& pointsOnEachCamera) - { - try - { - // Sanity checks - if (cameraMatrices.size() != pointsOnEachCamera.size()) - error("numberCameras.size() != pointsOnEachCamera.size() (" + std::to_string(cameraMatrices.size()) - + " vs. " + std::to_string(pointsOnEachCamera.size()) + ").", - __LINE__, __FUNCTION__, __FILE__); - if (cameraMatrices.empty()) - error("numberCameras.empty()", - __LINE__, __FUNCTION__, __FILE__); - // Create and fill A for homogenous equation system Ax = 0 - const auto numberCameras = (int)cameraMatrices.size(); - cv::Mat A = cv::Mat::zeros(numberCameras*2, 4, CV_64F); - for (auto i = 0 ; i < numberCameras ; i++) - { - A.rowRange(i*2, i*2+1) = pointsOnEachCamera[i].x*cameraMatrices[i].rowRange(2,3) - - cameraMatrices[i].rowRange(0,1); - A.rowRange(i*2+1, i*2+2) = pointsOnEachCamera[i].y*cameraMatrices[i].rowRange(2,3) - - cameraMatrices[i].rowRange(1,2); - } - // Solve x for Ax = 0 --> SVD on A - cv::SVD svd{A}; - svd.solveZ(A,reconstructedPoint); - reconstructedPoint /= reconstructedPoint.at(3); - - return calcReprojectionError(reconstructedPoint, cameraMatrices, pointsOnEachCamera); - } - catch (const std::exception& e) - { - error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return -1.; - } - } - - double triangulateWithOptimization( - cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, - const std::vector& pointsOnEachCamera, const double reprojectionMaxAcceptable) - { - try - { - // Warning - if (cameraMatrices.size() >= 8) - { - error("We did not have that many camera views to test the 3D triangulation code, so it might not" - " give the desired results here. But we would love to help! Please, share your video/images" - " so we can test our code with them and guarantee the desired results even for >= 8" - " cameras! Feel free to email them to gines@alumni.cmu.edu.", - __LINE__, __FUNCTION__, __FILE__); - } - - // Information for 3 cameras: - // - Speed: triangulate ~0.01 ms vs. optimization ~0.2 ms - // - Accuracy: initial reprojection error ~14-21, reduced ~5% with non-linear optimization - - // Basic triangulation - auto projectionError = triangulate(reconstructedPoint, cameraMatrices, pointsOnEachCamera); - - // Basic RANSAC (for >= 4 cameras if the reprojection error is higher than usual) - // 1. Run with all cameras (already done) - // 2. Run with all but 1 camera for each camera. - // 3. Use the one with minimum average reprojection error. - // Note: Meant to be used for up to 7-8 views. With more than that, it might not improve much. - // Set initial values - auto cameraMatricesFinal = cameraMatrices; - auto pointsOnEachCameraFinal = pointsOnEachCamera; - if (cameraMatrices.size() >= 4 - && projectionError > 0.5 * reprojectionMaxAcceptable - /*&& projectionError < 1.5 * reprojectionMaxAcceptable*/) - { - // Find best projection - auto bestReprojection = projectionError; - auto bestReprojectionIndex = -1; // -1 means with all camera views - cv::Mat bestReconstructedPoint; - for (auto i = 0u; i < cameraMatrices.size(); ++i) - { - // Set initial values - auto cameraMatricesSubset = cameraMatrices; - auto pointsOnEachCameraSubset = pointsOnEachCamera; - // Remove camera i - cameraMatricesSubset.erase(cameraMatricesSubset.begin() + i); - pointsOnEachCameraSubset.erase(pointsOnEachCameraSubset.begin() + i); - // Get new triangulation results - cv::Mat reconstructedPointSubset; - const auto projectionErrorSubset = triangulate( - reconstructedPointSubset, cameraMatricesSubset, pointsOnEachCameraSubset); - // If projection doesn't change much, this point is inlier (or all points are bad) - // Thus, save new best results only if considerably better - if (bestReprojection > projectionErrorSubset && projectionErrorSubset < 0.9*projectionError) - { - bestReprojection = projectionErrorSubset; - bestReprojectionIndex = i; - bestReconstructedPoint = reconstructedPointSubset; - } - } - // Remove noisy camera - if (bestReprojectionIndex != -1) // && bestReprojection < 0.5 * reprojectionMaxAcceptable) - { - // Remove camera i - cameraMatricesFinal.erase(cameraMatricesFinal.begin() + bestReprojectionIndex); - pointsOnEachCameraFinal.erase(pointsOnEachCameraFinal.begin() + bestReprojectionIndex); - // Update reconstructedPoint & projectionError - reconstructedPoint = bestReconstructedPoint; - projectionError = bestReprojection; - } - } - - #ifdef USE_CERES - // Empirically detected that reprojection error (for 4 cameras) only minimizes the error if initial - // project error > ~2.5, and that it improves more the higher that error actually is - // Therefore, we disable it for already accurate samples in order to get both: - // - Speed - // - Accuracy for already accurate samples - if (projectionError > 3.0 - && projectionError < 1.5*reprojectionMaxAcceptable) - { - // Slow equivalent: double paramX[3]; paramX[i] = reconstructedPoint.at(i); - double* paramX = (double*)reconstructedPoint.data; - ceres::Problem problem; - for (auto i = 0u; i < cameraMatricesFinal.size(); ++i) - { - // Slow copy equivalent: - // double camParam[12]; memcpy(camParam, cameraMatricesFinal[i].data, sizeof(double)*12); - const double* const camParam = (double*)cameraMatricesFinal[i].data; - // Each Residual block takes a point and a camera as input and outputs a 2 - // dimensional residual. Internally, the cost function stores the observed - // image location and compares the reprojection against the observation. - ceres::CostFunction* cost_function = - new ceres::AutoDiffCostFunction( - new ReprojectionErrorForTriangulation( - pointsOnEachCameraFinal[i].x, pointsOnEachCameraFinal[i].y, camParam)); - // Add to problem - problem.AddResidualBlock(cost_function, - //NULL, //squared loss - new ceres::HuberLoss(2.0), - paramX); // paramX[0,1,2] - } - - ceres::Solver::Options options; - options.linear_solver_type = ceres::DENSE_NORMAL_CHOLESKY; - // options.num_threads = 2; // It does not affect speed - // if (fastVersion) - { - // ~22 ms - // options.function_tolerance = 1e-3; //1e-6 - // options.gradient_tolerance = 1e-5; //1e-10 - // options.parameter_tolerance = 1e-5; //1e-8 - // options.inner_iteration_tolerance = 1e-3; //1e-6 - // ~30 ms (~30 FPS) - options.function_tolerance = 1e-4; //1e-6 - options.gradient_tolerance = 1e-7; //1e-10 - options.parameter_tolerance = 1e-6; //1e-8 - options.inner_iteration_tolerance = 1e-4; //1e-6 - // Default (none of the above) ~42 ms - } - // options.minimizer_progress_to_stdout = true; - // options.parameter_tolerance = 1e-20; - // options.function_tolerance = 1e-20; - ceres::Solver::Summary summary; - ceres::Solve(options, &problem, &summary); - // if (summary.initial_cost > summary.final_cost) - // std::cout << summary.FullReport() << "\n"; - - projectionError = calcReprojectionError(reconstructedPoint, cameraMatricesFinal, - pointsOnEachCameraFinal); - // const auto reprojectionErrorDecrease = std::sqrt((summary.initial_cost - summary.final_cost) - // / double(cameraMatricesFinal.size())); - } - #else - UNUSED(reprojectionMaxAcceptable); - #endif - // // This value is always 1 - // assert(reconstructedPoint.at(3) == 1.); - - // // Check that our implementation gives similar result than OpenCV - // // But we apply bundle adjustment + >2 views, so it should be better (and slower) than OpenCV one - // if (cameraMatricesFinal.size() == 4) - // { - // cv::Mat triangCoords4D; - // cv::triangulatePoints(cameraMatricesFinal.at(0), cameraMatricesFinal.at(3), - // std::vector{pointsOnEachCameraFinal.at(0)}, - // std::vector{pointsOnEachCameraFinal.at(3)}, triangCoords4D); - // triangCoords4D /= triangCoords4D.at(3); - // std::cout << reconstructedPoint << "\n" - // << triangCoords4D << "\n" - // << cv::norm(reconstructedPoint-triangCoords4D) << "\n" << std::endl; - // } - - return projectionError; - } - catch (const std::exception& e) - { - error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return -1.; - } - } - inline bool isValidKeypoint(const float* const keypointPtr, const Point& imageSize) { try @@ -500,7 +189,7 @@ namespace op } Array PoseTriangulation::reconstructArray( - const std::vector>& keypointsVector, const std::vector& cameraMatrices, + const std::vector>& keypointsVector, const std::vector& cameraMatrices, const std::vector>& imageSizes) const { try @@ -520,23 +209,24 @@ namespace op " (`--3d), you should also enable `--frame_undistort` so their camera parameters are read."}; std::vector> PoseTriangulation::reconstructArray( const std::vector>>& keypointsVectors, - const std::vector& cameraMatrices, + const std::vector& cameraMatrices, const std::vector>& imageSizes) const { try { + OP_OP2CVVECTORMAT(cvCameraMatrices, cameraMatrices); // Sanity checks - if (cameraMatrices.size() < 2) + if (cvCameraMatrices.size() < 2) error("3-D reconstruction (`--3d`) requires at least 2 camera views, only found " - + std::to_string(cameraMatrices.size()) + "camera parameter matrices." + sFlirErrorMessage, + + std::to_string(cvCameraMatrices.size()) + "camera parameter matrices." + sFlirErrorMessage, __LINE__, __FUNCTION__, __FILE__); - for (const auto& cameraMatrix : cameraMatrices) + for (const auto& cameraMatrix : cvCameraMatrices) if (cameraMatrix.empty()) error("Camera matrix was found empty during 3-D reconstruction (`--3d`)." + sFlirErrorMessage, __LINE__, __FUNCTION__, __FILE__); - if (cameraMatrices.size() != imageSizes.size()) + if (cvCameraMatrices.size() != imageSizes.size()) error("The camera parameters and number of images must be the same (" - + std::to_string(cameraMatrices.size()) + " vs. " + std::to_string(imageSizes.size()) + ").", + + std::to_string(cvCameraMatrices.size()) + " vs. " + std::to_string(imageSizes.size()) + ").", __LINE__, __FUNCTION__, __FILE__); // Run 3-D reconstruction bool keypointsReconstructed = false; @@ -547,14 +237,14 @@ namespace op // // Multi-thread option - ~15% slower // // Ceres seems to be super slow if run concurrently in different threads // threads.at(i) = std::thread{&reconstructArrayThread, - // &keypoints3Ds[i], keypointsVectors[i], cameraMatrices, + // &keypoints3Ds[i], keypointsVectors[i], cvCameraMatrices, // imageSizes, mMinViews3d}; // Single-thread option keypointsReconstructed |= reconstructArrayThread( - &keypoints3Ds[i], keypointsVectors[i], cameraMatrices, imageSizes, mMinViews3d); + &keypoints3Ds[i], keypointsVectors[i], cvCameraMatrices, imageSizes, mMinViews3d); } keypointsReconstructed |= reconstructArrayThread( - &keypoints3Ds.back(), keypointsVectors.back(), cameraMatrices, imageSizes, mMinViews3d); + &keypoints3Ds.back(), keypointsVectors.back(), cvCameraMatrices, imageSizes, mMinViews3d); // // Close threads // for (auto& thread : threads) // if (thread.joinable()) diff --git a/src/openpose/3d/poseTriangulationPrivate.cpp b/src/openpose/3d/poseTriangulationPrivate.cpp new file mode 100644 index 00000000..143be38c --- /dev/null +++ b/src/openpose/3d/poseTriangulationPrivate.cpp @@ -0,0 +1,317 @@ +#include +#ifdef USE_CERES + #include + #include +#endif +#include +#include + +namespace op +{ + double calcReprojectionError(const cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, + const std::vector& pointsOnEachCamera) + { + try + { + auto averageError = 0.; + for (auto i = 0u ; i < cameraMatrices.size() ; i++) + { + cv::Mat imageX = cameraMatrices[i] * reconstructedPoint; + imageX /= imageX.at(2,0); + const auto error = std::sqrt(std::pow(imageX.at(0,0) - pointsOnEachCamera[i].x,2) + + std::pow(imageX.at(1,0) - pointsOnEachCamera[i].y,2)); + // log("Error: " + std::to_string(error)); + averageError += error; + } + return averageError / cameraMatrices.size(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1.; + } + } + + #ifdef USE_CERES + // Nonlinear Optimization for 3D Triangulation + struct ReprojectionErrorForTriangulation + { + ReprojectionErrorForTriangulation(const double x, const double y, const double* const param) : + observed_x{x}, + observed_y{y} + { + memcpy(camParam, param, sizeof(double)*12); + } + + template + bool operator()(const T* const pt, + T* residuals) const ; + + inline virtual bool Evaluate(double const* const* pt, + double* residuals, + double** jacobians) const; + + const double observed_x; + const double observed_y; + double camParam[12]; + }; + + template + bool ReprojectionErrorForTriangulation::operator()(const T* const pt, + T* residuals) const + { + try + { + const T predicted[3] = { + T(camParam[0])*pt[0] + T(camParam[1])*pt[1] + T(camParam[2])*pt[2] + T(camParam[3]), + T(camParam[4])*pt[0] + T(camParam[5])*pt[1] + T(camParam[6])*pt[2] + T(camParam[7]), + T(camParam[8])*pt[0] + T(camParam[9])*pt[1] + T(camParam[10])*pt[2] + T(camParam[11])}; + + residuals[0] = T(observed_x) - predicted[0] / predicted[2]; + residuals[1] = T(observed_y) - predicted[1] / predicted[2]; + + // residuals[0] = T(pow(predicted[0] - observed_x,2) + pow(predicted[1] - observed_y,2)); + // residuals[0] = -pow(predicted[0] - T(observed_x),2); + // residuals[1] = -pow(predicted[1] - T(observed_y),2); + + return true; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return false; + } + } + + bool ReprojectionErrorForTriangulation::Evaluate(double const* const* pt, + double* residuals, + double** jacobians) const + { + try + { + UNUSED(jacobians); + + const double predicted[3] = { + camParam[0]*pt[0][0] + camParam[1]*pt[0][1] + camParam[2]*pt[0][2] + camParam[3], + camParam[4]*pt[0][0] + camParam[5]*pt[0][1] + camParam[6]*pt[0][2] + camParam[7], + camParam[8]*pt[0][0] + camParam[9]*pt[0][1] + camParam[10]*pt[0][2] + camParam[11]}; + + // residuals[0] = predicted[0] / predicted[2] - observed_x; + // residuals[1] = predicted[1] / predicted[2] - observed_y; + + residuals[0] = std::sqrt(std::pow(predicted[0] / predicted[2] - observed_x,2) + + std::pow(predicted[1] / predicted[2] - observed_y,2)); + + // log("Residuals:"); + // residuals[0]= pow(predicted[0] - (observed_x),2); + // residuals[1]= pow(predicted[1] - (observed_y),2); + + return true; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return false; + } + } + #endif + + double triangulate(cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, + const std::vector& pointsOnEachCamera) + { + try + { + // Sanity checks + if (cameraMatrices.size() != pointsOnEachCamera.size()) + error("numberCameras.size() != pointsOnEachCamera.size() (" + std::to_string(cameraMatrices.size()) + + " vs. " + std::to_string(pointsOnEachCamera.size()) + ").", + __LINE__, __FUNCTION__, __FILE__); + if (cameraMatrices.empty()) + error("numberCameras.empty()", + __LINE__, __FUNCTION__, __FILE__); + // Create and fill A for homogenous equation system Ax = 0 + const auto numberCameras = (int)cameraMatrices.size(); + cv::Mat A = cv::Mat::zeros(numberCameras*2, 4, CV_64F); + for (auto i = 0 ; i < numberCameras ; i++) + { + A.rowRange(i*2, i*2+1) = pointsOnEachCamera[i].x*cameraMatrices[i].rowRange(2,3) + - cameraMatrices[i].rowRange(0,1); + A.rowRange(i*2+1, i*2+2) = pointsOnEachCamera[i].y*cameraMatrices[i].rowRange(2,3) + - cameraMatrices[i].rowRange(1,2); + } + // Solve x for Ax = 0 --> SVD on A + cv::SVD svd{A}; + svd.solveZ(A,reconstructedPoint); + reconstructedPoint /= reconstructedPoint.at(3); + + return calcReprojectionError(reconstructedPoint, cameraMatrices, pointsOnEachCamera); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1.; + } + } + + double triangulateWithOptimization( + cv::Mat& reconstructedPoint, const std::vector& cameraMatrices, + const std::vector& pointsOnEachCamera, const double reprojectionMaxAcceptable) + { + try + { + // Warning + if (cameraMatrices.size() >= 8) + { + error("We did not have that many camera views to test the 3D triangulation code, so it might not" + " give the desired results here. But we would love to help! Please, share your video/images" + " so we can test our code with them and guarantee the desired results even for >= 8" + " cameras! Feel free to email them to gines@alumni.cmu.edu.", + __LINE__, __FUNCTION__, __FILE__); + } + + // Information for 3 cameras: + // - Speed: triangulate ~0.01 ms vs. optimization ~0.2 ms + // - Accuracy: initial reprojection error ~14-21, reduced ~5% with non-linear optimization + + // Basic triangulation + auto projectionError = triangulate(reconstructedPoint, cameraMatrices, pointsOnEachCamera); + + // Basic RANSAC (for >= 4 cameras if the reprojection error is higher than usual) + // 1. Run with all cameras (already done) + // 2. Run with all but 1 camera for each camera. + // 3. Use the one with minimum average reprojection error. + // Note: Meant to be used for up to 7-8 views. With more than that, it might not improve much. + // Set initial values + auto cameraMatricesFinal = cameraMatrices; + auto pointsOnEachCameraFinal = pointsOnEachCamera; + if (cameraMatrices.size() >= 4 + && projectionError > 0.5 * reprojectionMaxAcceptable + /*&& projectionError < 1.5 * reprojectionMaxAcceptable*/) + { + // Find best projection + auto bestReprojection = projectionError; + auto bestReprojectionIndex = -1; // -1 means with all camera views + cv::Mat bestReconstructedPoint; + for (auto i = 0u; i < cameraMatrices.size(); ++i) + { + // Set initial values + auto cameraMatricesSubset = cameraMatrices; + auto pointsOnEachCameraSubset = pointsOnEachCamera; + // Remove camera i + cameraMatricesSubset.erase(cameraMatricesSubset.begin() + i); + pointsOnEachCameraSubset.erase(pointsOnEachCameraSubset.begin() + i); + // Get new triangulation results + cv::Mat reconstructedPointSubset; + const auto projectionErrorSubset = triangulate( + reconstructedPointSubset, cameraMatricesSubset, pointsOnEachCameraSubset); + // If projection doesn't change much, this point is inlier (or all points are bad) + // Thus, save new best results only if considerably better + if (bestReprojection > projectionErrorSubset && projectionErrorSubset < 0.9*projectionError) + { + bestReprojection = projectionErrorSubset; + bestReprojectionIndex = i; + bestReconstructedPoint = reconstructedPointSubset; + } + } + // Remove noisy camera + if (bestReprojectionIndex != -1) // && bestReprojection < 0.5 * reprojectionMaxAcceptable) + { + // Remove camera i + cameraMatricesFinal.erase(cameraMatricesFinal.begin() + bestReprojectionIndex); + pointsOnEachCameraFinal.erase(pointsOnEachCameraFinal.begin() + bestReprojectionIndex); + // Update reconstructedPoint & projectionError + reconstructedPoint = bestReconstructedPoint; + projectionError = bestReprojection; + } + } + + #ifdef USE_CERES + // Empirically detected that reprojection error (for 4 cameras) only minimizes the error if initial + // project error > ~2.5, and that it improves more the higher that error actually is + // Therefore, we disable it for already accurate samples in order to get both: + // - Speed + // - Accuracy for already accurate samples + if (projectionError > 3.0 + && projectionError < 1.5*reprojectionMaxAcceptable) + { + // Slow equivalent: double paramX[3]; paramX[i] = reconstructedPoint.at(i); + double* paramX = (double*)reconstructedPoint.data; + ceres::Problem problem; + for (auto i = 0u; i < cameraMatricesFinal.size(); ++i) + { + // Slow copy equivalent: + // double camParam[12]; memcpy(camParam, cameraMatricesFinal[i].data, sizeof(double)*12); + const double* const camParam = (double*)cameraMatricesFinal[i].data; + // Each Residual block takes a point and a camera as input and outputs a 2 + // dimensional residual. Internally, the cost function stores the observed + // image location and compares the reprojection against the observation. + ceres::CostFunction* cost_function = + new ceres::AutoDiffCostFunction( + new ReprojectionErrorForTriangulation( + pointsOnEachCameraFinal[i].x, pointsOnEachCameraFinal[i].y, camParam)); + // Add to problem + problem.AddResidualBlock(cost_function, + //NULL, //squared loss + new ceres::HuberLoss(2.0), + paramX); // paramX[0,1,2] + } + + ceres::Solver::Options options; + options.linear_solver_type = ceres::DENSE_NORMAL_CHOLESKY; + // options.num_threads = 2; // It does not affect speed + // if (fastVersion) + { + // ~22 ms + // options.function_tolerance = 1e-3; //1e-6 + // options.gradient_tolerance = 1e-5; //1e-10 + // options.parameter_tolerance = 1e-5; //1e-8 + // options.inner_iteration_tolerance = 1e-3; //1e-6 + // ~30 ms (~30 FPS) + options.function_tolerance = 1e-4; //1e-6 + options.gradient_tolerance = 1e-7; //1e-10 + options.parameter_tolerance = 1e-6; //1e-8 + options.inner_iteration_tolerance = 1e-4; //1e-6 + // Default (none of the above) ~42 ms + } + // options.minimizer_progress_to_stdout = true; + // options.parameter_tolerance = 1e-20; + // options.function_tolerance = 1e-20; + ceres::Solver::Summary summary; + ceres::Solve(options, &problem, &summary); + // if (summary.initial_cost > summary.final_cost) + // std::cout << summary.FullReport() << "\n"; + + projectionError = calcReprojectionError(reconstructedPoint, cameraMatricesFinal, + pointsOnEachCameraFinal); + // const auto reprojectionErrorDecrease = std::sqrt((summary.initial_cost - summary.final_cost) + // / double(cameraMatricesFinal.size())); + } + #else + UNUSED(reprojectionMaxAcceptable); + #endif + // // This value is always 1 + // assert(reconstructedPoint.at(3) == 1.); + + // // Check that our implementation gives similar result than OpenCV + // // But we apply bundle adjustment + >2 views, so it should be better (and slower) than OpenCV one + // if (cameraMatricesFinal.size() == 4) + // { + // cv::Mat triangCoords4D; + // cv::triangulatePoints(cameraMatricesFinal.at(0), cameraMatricesFinal.at(3), + // std::vector{pointsOnEachCameraFinal.at(0)}, + // std::vector{pointsOnEachCameraFinal.at(3)}, triangCoords4D); + // triangCoords4D /= triangCoords4D.at(3); + // std::cout << reconstructedPoint << "\n" + // << triangCoords4D << "\n" + // << cv::norm(reconstructedPoint-triangCoords4D) << "\n" << std::endl; + // } + + return projectionError; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1.; + } + } +} diff --git a/src/openpose/calibration/cameraParameterEstimation.cpp b/src/openpose/calibration/cameraParameterEstimation.cpp index f7603508..2cdbdf71 100644 --- a/src/openpose/calibration/cameraParameterEstimation.cpp +++ b/src/openpose/calibration/cameraParameterEstimation.cpp @@ -1,21 +1,21 @@ +#include #include #include // std::accumulate #ifdef USE_CERES #include #include #endif -#include #ifdef USE_EIGEN #include #include #endif #include -#include -#include #include #include #include -#include +#include +#include +#include namespace op { @@ -831,7 +831,10 @@ namespace op fileRemoved = {remove(finalPath.c_str()) == 0}; // save images on hhd in the desired place if (i < imagesWithCorners.size()) - saveImage(imagesWithCorners.at(i), finalPath); + { + const auto opMat = OP_CV2OPMAT(imagesWithCorners.at(i)); + saveImage(opMat, finalPath); + } } } } @@ -923,8 +926,10 @@ namespace op // Save intrinsics/results log("", Priority::Low, __LINE__, __FUNCTION__, __FILE__); + Matrix opCameraMatrix = OP_CV2OPMAT(intrinsics.cameraMatrix); + Matrix opDistortionCoefficients = OP_CV2OPMAT(intrinsics.distortionCoefficients); CameraParameterReader cameraParameterReader{ - serialNumber, intrinsics.cameraMatrix, intrinsics.distortionCoefficients}; + serialNumber, opCameraMatrix, opDistortionCoefficients }; cameraParameterReader.writeParameters(outputParameterFolder); // Debugging (optional) - Save images with corners @@ -946,7 +951,10 @@ namespace op fileRemoved = {remove(finalPath.c_str()) == 0}; // save images on hhd in the desired place if (i < imagesWithCorners.size()) - saveImage(imagesWithCorners.at(i), finalPath); + { + const auto opMat = OP_CV2OPMAT(imagesWithCorners.at(i)); + saveImage(opMat, finalPath); + } } } } @@ -978,8 +986,10 @@ namespace op CameraParameterReader cameraParameterReader; cameraParameterReader.readParameters(parameterFolder); const auto cameraSerialNumbers = cameraParameterReader.getCameraSerialNumbers(); - const auto realCameraDistortions = cameraParameterReader.getCameraDistortions(); - auto cameraIntrinsicsSubset = cameraParameterReader.getCameraIntrinsics(); + const auto opRealCameraDistortions = cameraParameterReader.getCameraDistortions(); + OP_OP2CVVECTORMAT(realCameraDistortions, opRealCameraDistortions) + auto opCameraIntrinsicsSubset = cameraParameterReader.getCameraIntrinsics(); + OP_OP2CVVECTORMAT(cameraIntrinsicsSubset, opCameraIntrinsicsSubset) auto cameraDistortionsSubset = (imagesAreUndistorted ? std::vector{realCameraDistortions.size()} : realCameraDistortions); @@ -993,7 +1003,9 @@ namespace op bool cam0IsOrigin = true; if (combineCam0Extrinsics) { - cameraParameterReader.getCameraExtrinsics().at(index0).copyTo(extrinsicsCam0(cv::Rect{0,0,4,3})); + const cv::Mat cameraExtrinsicsAtIndex0 = OP_OP2CVCONSTMAT( + cameraParameterReader.getCameraExtrinsics().at(index0)); + cameraExtrinsicsAtIndex0.copyTo(extrinsicsCam0(cv::Rect{0,0,4,3})); cam0IsOrigin = cv::norm(extrinsicsCam0 - cv::Mat::eye(4, 4, extrinsicsCam0.type())) < 1e-9; } @@ -1173,9 +1185,10 @@ namespace op log("", Priority::Low, __LINE__, __FUNCTION__, __FILE__); CameraParameterReader cameraParameterReaderFinal{ cameraSerialNumbers.at(index1), - cameraIntrinsicsSubset.at(1), - realCameraDistortions.at(index1), - cvMatExtrinsics}; + OP_CV2OPMAT(cameraIntrinsicsSubset.at(1)), + OP_CV2OPMAT(realCameraDistortions.at(index1)), + OP_CV2OPMAT(cvMatExtrinsics) + }; cameraParameterReaderFinal.writeParameters(parameterFolder); // Let the rendered image to be displayed @@ -2117,14 +2130,14 @@ namespace op log("Loading parameters...", Priority::High); CameraParameterReader cameraParameterReader; cameraParameterReader.readParameters(parameterFolder); - const auto cameraExtrinsicsInitial = cameraParameterReader.getCameraExtrinsicsInitial(); + const auto opCameraExtrinsicsInitial = cameraParameterReader.getCameraExtrinsicsInitial(); // Sanity check - if (cameraExtrinsicsInitial.empty()) + if (opCameraExtrinsicsInitial.empty()) error("Camera intrinsics could not be loaded from " + parameterFolder + ". Are they in the right path? Remember than the XML must contain the right intrinsic" + " parameters before using this function. ", __LINE__, __FUNCTION__, __FILE__); bool initialEmpty = false; - for (const auto& cameraExtrinsicInitial : cameraExtrinsicsInitial) + for (const auto& cameraExtrinsicInitial : opCameraExtrinsicsInitial) { if (cameraExtrinsicInitial.empty()) { @@ -2135,16 +2148,18 @@ namespace op log("Parameters loaded.", Priority::High); // Camera extrinsics log("", Priority::Low, __LINE__, __FUNCTION__, __FILE__); - auto cameraExtrinsics = (initialEmpty - ? cameraParameterReader.getCameraExtrinsics() : cameraExtrinsicsInitial); + auto opCameraExtrinsics = (initialEmpty + ? cameraParameterReader.getCameraExtrinsics() : opCameraExtrinsicsInitial); + OP_OP2CVVECTORMAT(cameraExtrinsics, opCameraExtrinsics) // The first one should be [I | 0]: Multiply them all by inv(camera 0 extrinsics) cv::Mat cameraOriginInv; cameraXAsOrigin(cameraExtrinsics, cameraOriginInv, cameraExtrinsics.at(0).clone()); // Camera intrinsics and distortion - const auto cameraIntrinsics = cameraParameterReader.getCameraIntrinsics(); + const auto opCameraIntrinsics = cameraParameterReader.getCameraIntrinsics(); + OP_OP2CVVECTORMAT(cameraIntrinsics, opCameraIntrinsics); const auto cameraDistortions = ( imagesAreUndistorted - ? std::vector{cameraIntrinsics.size()} : cameraParameterReader.getCameraDistortions()); + ? std::vector{cameraIntrinsics.size()} : cameraParameterReader.getCameraDistortions()); // Read images in folder log("Reading images in folder...", Priority::High); const auto numberCorners = gridInnerCorners.area(); @@ -2156,7 +2171,7 @@ namespace op // Get 2D grid corners of each image std::vector imagesWithCorners; const auto imageSize = imageAndPaths.at(0).first.size(); - const auto numberViews = imageAndPaths.size() / numberCameras; + const auto numberViews = (unsigned int)(imageAndPaths.size() / numberCameras); log("Processing cameras...", Priority::High); std::vector threads; for (auto cameraIndex = 0 ; cameraIndex < numberCameras ; cameraIndex++) @@ -2292,15 +2307,15 @@ namespace op // Save new extrinsics log("", Priority::Low, __LINE__, __FUNCTION__, __FILE__); const auto cameraSerialNumbers = cameraParameterReader.getCameraSerialNumbers(); - const auto realCameraDistortions = cameraParameterReader.getCameraDistortions(); + const auto opRealCameraDistortions = cameraParameterReader.getCameraDistortions(); for (auto i = 0 ; i < numberCameras ; i++) { CameraParameterReader cameraParameterReaderFinal{ cameraSerialNumbers.at(i), - cameraIntrinsics.at(i), - realCameraDistortions.at(i), - refinedExtrinsics.at(i), - (initialEmpty ? cameraExtrinsics.at(i) : cameraExtrinsicsInitial.at(i))}; + OP_CV2OPCONSTMAT(cameraIntrinsics.at(i)), + opRealCameraDistortions.at(i), + OP_CV2OPCONSTMAT(refinedExtrinsics.at(i)), + (initialEmpty ? OP_CV2OPCONSTMAT(cameraExtrinsics.at(i)) : opCameraExtrinsicsInitial.at(i))}; cameraParameterReaderFinal.writeParameters(parameterFolder); } log(" ", Priority::High); @@ -2344,7 +2359,7 @@ namespace op // Get 2D grid corners of each image std::vector imagesWithCorners; const auto imageSize = imageAndPaths.at(0).first.size(); - const auto numberViews = imageAndPaths.size() / numberCameras; + const auto numberViews = (unsigned int)(imageAndPaths.size() / numberCameras); log("Processing cameras...", Priority::High); std::vector threads; for (auto cameraIndex = 0 ; cameraIndex < numberCameras ; cameraIndex++) diff --git a/src/openpose/calibration/gridPatternFunctions.cpp b/src/openpose/calibration/gridPatternFunctions.cpp index 9e5fa5f4..0dfffb6d 100644 --- a/src/openpose/calibration/gridPatternFunctions.cpp +++ b/src/openpose/calibration/gridPatternFunctions.cpp @@ -1,6 +1,7 @@ +#include #include #include -#include +#include namespace op { diff --git a/src/openpose/core/CMakeLists.txt b/src/openpose/core/CMakeLists.txt index 2ddcc43a..92cfc26c 100644 --- a/src/openpose/core/CMakeLists.txt +++ b/src/openpose/core/CMakeLists.txt @@ -9,6 +9,7 @@ set(SOURCES_OP_CORE gpuRenderer.cpp keepTopNPeople.cpp keypointScaler.cpp + matrix.cpp opOutputToCvMat.cpp point.cpp rectangle.cpp diff --git a/src/openpose/core/array.cpp b/src/openpose/core/array.cpp index e7cf578c..6d2c2a0d 100644 --- a/src/openpose/core/array.cpp +++ b/src/openpose/core/array.cpp @@ -1,7 +1,8 @@ +#include #include // typeid #include // std::accumulate -#include -#include +#include // cv::Mat +#include // Note: std::shared_ptr not (fully) supported for array pointers: // http://stackoverflow.com/questions/8947579/ @@ -21,12 +22,12 @@ namespace op * std::shared_ptr points to. */ template - void setCvMatFromPtr(std::pair& cvMatData, T* const dataPtr, const std::vector& sizes) + void setCvMatFromPtr(std::pair& cvMatData, T* const dataPtr, const std::vector& sizes) { try { cvMatData.first = true; - cvMatData.second = cv::Mat(); + cvMatData.second = Matrix(); // BGR image if (sizes.size() == 3 && sizes[2] == 3) { @@ -46,7 +47,10 @@ namespace op cvMatData.first = false; if (cvMatData.first) - cvMatData.second = cv::Mat(sizes[0], sizes[1], cvFormat, dataPtr); + { + cv::Mat cvMat(sizes[0], sizes[1], cvFormat, dataPtr); + cvMatData.second = OP_CV2OPMAT(cvMat); + } } // Any other type else @@ -67,7 +71,10 @@ namespace op cvMatData.first = false; if (cvMatData.first) - cvMatData.second = cv::Mat((int)sizes.size(), sizes.data(), cvFormat, dataPtr); + { + cv::Mat cvMat((int)sizes.size(), sizes.data(), cvFormat, dataPtr); + cvMatData.second = OP_CV2OPMAT(cvMat); + } } } catch (const std::exception& e) @@ -362,16 +369,16 @@ namespace op } template - void Array::setFrom(const cv::Mat& cvMat) + void Array::setFrom(const Matrix& cvMat) { try { if (!cvMat.empty()) { // New size - std::vector newSize(cvMat.dims,0); + std::vector newSize(cvMat.dims(),0); for (auto i = 0u ; i < newSize.size() ; i++) - newSize[i] = cvMat.size[i]; + newSize[i] = cvMat.size(i); // Reset data & volume reset(newSize); // Integrity checks @@ -528,12 +535,12 @@ namespace op } template - const cv::Mat& Array::getConstCvMat() const + const Matrix& Array::getConstCvMat() const { try { if (!mCvMatData.first) - error("Array: cv::Mat functions only valid for T types defined by OpenCV: unsigned char," + error("Array: Matrix functions only valid for T types defined by OpenCV: unsigned char," " signed char, int, float & double", __LINE__, __FUNCTION__, __FILE__); return mCvMatData.second; } @@ -545,12 +552,12 @@ namespace op } template - cv::Mat& Array::getCvMat() + Matrix& Array::getCvMat() { try { if (!mCvMatData.first) - error("Array: cv::Mat functions only valid for T types defined by OpenCV: unsigned char," + error("Array: Matrix functions only valid for T types defined by OpenCV: unsigned char," " signed char, int, float & double", __LINE__, __FUNCTION__, __FILE__); return mCvMatData.second; } @@ -687,8 +694,8 @@ namespace op mVolume = 0ul; spData.reset(); pData = nullptr; - // cv::Mat available but empty - mCvMatData = std::make_pair(true, cv::Mat()); + // Matrix available but empty + mCvMatData = std::make_pair(true, Matrix()); } } catch (const std::exception& e) diff --git a/src/openpose/core/cvMatToOpInput.cpp b/src/openpose/core/cvMatToOpInput.cpp index 846d36c3..0076dcf9 100644 --- a/src/openpose/core/cvMatToOpInput.cpp +++ b/src/openpose/core/cvMatToOpInput.cpp @@ -1,12 +1,12 @@ -// #include +#include #ifdef USE_CUDA #include - #include #include + #include #endif #include #include -#include +#include namespace op { @@ -61,21 +61,22 @@ namespace op } std::vector> CvMatToOpInput::createArray( - const cv::Mat& cvInputData, const std::vector& scaleInputToNetInputs, + const Matrix& inputData, const std::vector& scaleInputToNetInputs, const std::vector>& netInputSizes) { try { // Sanity checks - if (cvInputData.empty()) - error("Wrong input element (empty cvInputData).", __LINE__, __FUNCTION__, __FILE__); - if (cvInputData.channels() != 3) + if (inputData.empty()) + error("Wrong input element (empty inputData).", __LINE__, __FUNCTION__, __FILE__); + if (inputData.channels() != 3) error("Input images must be 3-channel BGR.", __LINE__, __FUNCTION__, __FILE__); if (scaleInputToNetInputs.size() != netInputSizes.size()) error("scaleInputToNetInputs.size() != netInputSizes.size().", __LINE__, __FUNCTION__, __FILE__); // inputNetData - Reescale keeping aspect ratio and transform to float the input deep net image const auto numberScales = (int)scaleInputToNetInputs.size(); std::vector> inputNetData(numberScales); + cv::Mat cvInputData = OP_OP2CVCONSTMAT(inputData); for (auto i = 0u ; i < inputNetData.size() ; i++) { // CPU version (faster if #Gpus <= 3 and relatively small images) @@ -86,7 +87,8 @@ namespace op // Fill inputNetData[i] inputNetData[i].reset({1, 3, netInputSizes.at(i).y, netInputSizes.at(i).x}); uCharCvMatToFloatPtr( - inputNetData[i].getPtr(), frameWithNetSize, (mPoseModel == PoseModel::BODY_19N ? 2 : 1)); + inputNetData[i].getPtr(), OP_CV2OPMAT(frameWithNetSize), + (mPoseModel == PoseModel::BODY_19N ? 2 : 1)); // // OpenCV equivalent // const auto scale = 1/255.; diff --git a/src/openpose/core/cvMatToOpOutput.cpp b/src/openpose/core/cvMatToOpOutput.cpp index d2fb7de6..57b0d3f1 100644 --- a/src/openpose/core/cvMatToOpOutput.cpp +++ b/src/openpose/core/cvMatToOpOutput.cpp @@ -1,10 +1,10 @@ +#include #ifdef USE_CUDA #include - #include #include + #include #endif -#include -#include +#include namespace op { @@ -74,10 +74,11 @@ namespace op } Array CvMatToOpOutput::createArray( - const cv::Mat& cvInputData, const double scaleInputToOutput, const Point& outputResolution) + const Matrix& inputData, const double scaleInputToOutput, const Point& outputResolution) { try { + cv::Mat cvInputData = OP_OP2CVCONSTMAT(inputData); // Sanity checks if (cvInputData.empty()) error("Wrong input element (empty cvInputData).", __LINE__, __FUNCTION__, __FILE__); @@ -94,7 +95,9 @@ namespace op { cv::Mat frameWithOutputSize; resizeFixedAspectRatio(frameWithOutputSize, cvInputData, scaleInputToOutput, outputResolution); - frameWithOutputSize.convertTo(outputData.getCvMat(), CV_32FC3); + // Equivalent: frameWithOutputSize.convertTo(outputData.getCvMat(), CV_32FC3); + cv::Mat cvOutputData = OP_OP2CVMAT(outputData.getCvMat()); + frameWithOutputSize.convertTo(cvOutputData, CV_32FC3); } // CUDA version (if #Gpus > 3) else diff --git a/src/openpose/core/keepTopNPeople.cpp b/src/openpose/core/keepTopNPeople.cpp index 7598c756..566558d6 100644 --- a/src/openpose/core/keepTopNPeople.cpp +++ b/src/openpose/core/keepTopNPeople.cpp @@ -1,5 +1,7 @@ -#include #include +#include // std::sort +#include // std::sqrt +#include namespace op { @@ -45,7 +47,7 @@ namespace op numberPeopleAboveThreshold++; // Remove extra people - Fille topPeopleArray - // assignedPeopleOnThreshold avoids that people with repeated threshold remove higher elements. + // assignedPeopleOnThreshold avoids that people with repeated threshold remove higher elements. // In our case, it will keep the first N people with score = threshold, while keeping all the // people with higher scores. // E.g., poseFinalScores = [0, 0.5, 0.5, 0.5, 1.0]; mNumberPeopleMax = 2 diff --git a/src/openpose/core/matrix.cpp b/src/openpose/core/matrix.cpp new file mode 100644 index 00000000..507d2632 --- /dev/null +++ b/src/openpose/core/matrix.cpp @@ -0,0 +1,341 @@ +#include +#include // cv::Mat +#include + +namespace op +{ + struct Matrix::ImplMat + { + cv::Mat mCvMat; + }; + + Matrix::Matrix() : + spImpl{std::make_shared()} + { + } + + Matrix::Matrix(const void* cvMatPtr) : + spImpl{std::make_shared()} + { + try + { + spImpl->mCvMat = *((cv::Mat*) cvMatPtr); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } + + Matrix::Matrix(const int rows, const int cols, const int type, void* cvMatPtr) : + spImpl{std::make_shared()} + { + try + { + spImpl->mCvMat = cv::Mat(rows, cols, type, cvMatPtr); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } + + Matrix Matrix::clone() const + { + try + { + Matrix matrix; + matrix.spImpl->mCvMat = spImpl->mCvMat.clone(); + return matrix; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return Matrix(); + } + } + + void* Matrix::getCvMat() + { + try + { + return (void*)(&spImpl->mCvMat); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } + + const void* Matrix::getConstCvMat() const + { + try + { + return (const void* const)(&spImpl->mCvMat); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } + + unsigned char* Matrix::data() + { + try + { + return spImpl->mCvMat.data; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return nullptr; + } + } + + const unsigned char* Matrix::dataConst() const + { + try + { + return spImpl->mCvMat.data; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return nullptr; + } + } + + Matrix Matrix::eye(const int rows, const int cols, const int type) + { + try + { + Matrix matrix; + matrix.spImpl->mCvMat = cv::Mat::eye(rows, cols, type); + return matrix; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return Matrix(); + } + } + + int Matrix::cols() const + { + try + { + return spImpl->mCvMat.cols; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int Matrix::rows() const + { + try + { + return spImpl->mCvMat.rows; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int Matrix::size(const int dimension) const + { + try + { + return spImpl->mCvMat.size[dimension]; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int Matrix::dims() const + { + try + { + return spImpl->mCvMat.dims; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + bool Matrix::isContinuous() const + { + try + { + return spImpl->mCvMat.isContinuous(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return false; + } + } + + bool Matrix::isSubmatrix() const + { + try + { + return spImpl->mCvMat.isSubmatrix(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return false; + } + } + size_t Matrix::elemSize() const + { + try + { + return spImpl->mCvMat.elemSize(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return size_t(-1); + } + } + + size_t Matrix::elemSize1() const + { + try + { + return spImpl->mCvMat.elemSize(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return size_t(-1); + } + } + + int Matrix::type() const + { + try + { + return spImpl->mCvMat.type(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int Matrix::depth() const + { + try + { + return spImpl->mCvMat.depth(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + + int Matrix::channels() const + { + try + { + return spImpl->mCvMat.channels(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + size_t Matrix::step1(const int i) const + { + try + { + return spImpl->mCvMat.step1(i); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return size_t(-1); + } + } + + bool Matrix::empty() const + { + try + { + return spImpl->mCvMat.empty(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return false; + } + } + + size_t Matrix::total() const + { + try + { + return spImpl->mCvMat.total(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return size_t(-1); + } + } + + int Matrix::checkVector(const int elemChannels, const int depth, const bool requireContinuous) const + { + try + { + return spImpl->mCvMat.checkVector(elemChannels, depth, requireContinuous); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + void Matrix::setTo(const double value) + { + try + { + spImpl->mCvMat.setTo(value); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } + + void Matrix::copyTo(Matrix& outputMat) const + { + try + { + spImpl->mCvMat.copyTo(outputMat.spImpl->mCvMat); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } +} diff --git a/src/openpose/core/opOutputToCvMat.cpp b/src/openpose/core/opOutputToCvMat.cpp index b351c689..673cbbd3 100644 --- a/src/openpose/core/opOutputToCvMat.cpp +++ b/src/openpose/core/opOutputToCvMat.cpp @@ -1,9 +1,10 @@ +#include +#include // cv::Mat #ifdef USE_CUDA #include - #include + #include #endif #include -#include namespace op { @@ -73,7 +74,7 @@ namespace op } } - cv::Mat OpOutputToCvMat::formatToCvMat(const Array& outputData) + Matrix OpOutputToCvMat::formatToCvMat(const Array& outputData) { try { @@ -86,7 +87,9 @@ namespace op if (!mGpuResize) { // outputData to cvMat - outputData.getConstCvMat().convertTo(cvMat, CV_8UC3); + // Equivalent: outputData.getConstCvMat().convertTo(cvMat, CV_8UC3); + const cv::Mat constCvMat = OP_OP2CVCONSTMAT(outputData.getConstCvMat()); + constCvMat.convertTo(cvMat, CV_8UC3); } // CUDA version else @@ -116,12 +119,13 @@ namespace op #endif } // Return cvMat - return cvMat; + const Matrix opMat = OP_CV2OPMAT(cvMat); + return opMat; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } } diff --git a/src/openpose/face/CMakeLists.txt b/src/openpose/face/CMakeLists.txt index e19f3fba..71da8c6d 100644 --- a/src/openpose/face/CMakeLists.txt +++ b/src/openpose/face/CMakeLists.txt @@ -23,12 +23,12 @@ if (UNIX OR APPLE) endif () target_link_libraries(openpose_face openpose_core) - + if (BUILD_CAFFE) add_dependencies(openpose_face openpose) endif (BUILD_CAFFE) - install(TARGETS openpose_face + install(TARGETS openpose_face EXPORT OpenPose RUNTIME DESTINATION bin LIBRARY DESTINATION lib diff --git a/src/openpose/face/faceDetector.cpp b/src/openpose/face/faceDetector.cpp index 6da64cd8..2e9cecd6 100644 --- a/src/openpose/face/faceDetector.cpp +++ b/src/openpose/face/faceDetector.cpp @@ -2,7 +2,7 @@ #include #include #include - + namespace op { FaceDetector::FaceDetector(const PoseModel poseModel) : diff --git a/src/openpose/face/faceDetectorOpenCV.cpp b/src/openpose/face/faceDetectorOpenCV.cpp index 9d03eef0..4fd26888 100644 --- a/src/openpose/face/faceDetectorOpenCV.cpp +++ b/src/openpose/face/faceDetectorOpenCV.cpp @@ -1,15 +1,22 @@ -#include // cv::COLOR_BGR2GRAY -#include #include +#include // cv::CascadeClassifier +#include +#include namespace op { - FaceDetectorOpenCV::FaceDetectorOpenCV(const std::string& modelFolder) + struct FaceDetectorOpenCV::ImplFaceDetectorOpenCV + { + cv::CascadeClassifier mFaceCascade; + }; + + FaceDetectorOpenCV::FaceDetectorOpenCV(const std::string& modelFolder) : + upImpl{new ImplFaceDetectorOpenCV{}} { try { const std::string faceDetectorModelPath{modelFolder + "face/haarcascade_frontalface_alt.xml"}; - if (!mFaceCascade.load(faceDetectorModelPath)) + if (!upImpl->mFaceCascade.load(faceDetectorModelPath)) error("Face detector model not found at: " + faceDetectorModelPath, __LINE__, __FUNCTION__, __FILE__); } catch (const std::exception& e) @@ -22,10 +29,11 @@ namespace op { } - std::vector> FaceDetectorOpenCV::detectFaces(const cv::Mat& cvInputData) + std::vector> FaceDetectorOpenCV::detectFaces(const Matrix& inputData) { try { + cv::Mat cvInputData = OP_OP2CVCONSTMAT(inputData); // Image to grey and pyrDown cv::Mat frameGray; cv::cvtColor(cvInputData, frameGray, cv::COLOR_BGR2GRAY); @@ -38,7 +46,7 @@ namespace op // Face detection - Example from: // http://docs.opencv.org/2.4/doc/tutorials/objdetect/cascade_classifier/cascade_classifier.html std::vector detectedFaces; - mFaceCascade.detectMultiScale(frameGray, detectedFaces, 1.2, 3, 0|CV_HAAR_SCALE_IMAGE); + upImpl->mFaceCascade.detectMultiScale(frameGray, detectedFaces, 1.2, 3, 0|CV_HAAR_SCALE_IMAGE); // Rescale rectangles std::vector> faceRectangles(detectedFaces.size()); for(auto i = 0u; i < detectedFaces.size(); i++) diff --git a/src/openpose/face/faceExtractorCaffe.cpp b/src/openpose/face/faceExtractorCaffe.cpp index 8317a819..9112ec2d 100644 --- a/src/openpose/face/faceExtractorCaffe.cpp +++ b/src/openpose/face/faceExtractorCaffe.cpp @@ -1,7 +1,7 @@ +#include #ifdef USE_CAFFE #include #endif -#include // CV_WARP_INVERSE_MAP, CV_INTER_LINEAR #include #include #include @@ -9,7 +9,7 @@ #include #include #include -#include +#include namespace op { @@ -49,7 +49,7 @@ namespace op const auto volumeBodyParts = FACE_NUMBER_PARTS * channelOffset; auto totalOffset = 0u; auto* heatMapsPtr = &heatMaps.getPtr()[person*volumeBodyParts]; - // Copy face parts + // Copy face parts #ifdef USE_CUDA cudaMemcpy(heatMapsPtr, heatMapsGpuPtr, volumeBodyParts * sizeof(float), cudaMemcpyDeviceToHost); #else @@ -172,13 +172,15 @@ namespace op } void FaceExtractorCaffe::forwardPass( - const std::vector>& faceRectangles, const cv::Mat& cvInputData) + const std::vector>& faceRectangles, const Matrix& inputData) { try { #ifdef USE_CAFFE if (mEnabled && !faceRectangles.empty()) { + const cv::Mat cvInputData = OP_OP2CVCONSTMAT(inputData); + // Sanity check if (cvInputData.empty()) error("Empty cvInputData.", __LINE__, __FUNCTION__, __FILE__); @@ -240,7 +242,7 @@ namespace op cv::BORDER_CONSTANT, cv::Scalar(0,0,0)); // cv::Mat -> float* - uCharCvMatToFloatPtr(mFaceImageCrop.getPtr(), faceImage, true); + uCharCvMatToFloatPtr(mFaceImageCrop.getPtr(), OP_CV2OPMAT(faceImage), true); // // Debugging // if (person < 5) diff --git a/src/openpose/face/renderFace.cu b/src/openpose/face/renderFace.cu index 34b71776..834e35a9 100644 --- a/src/openpose/face/renderFace.cu +++ b/src/openpose/face/renderFace.cu @@ -1,8 +1,8 @@ +#include #include #include -#include -#include -#include +#include +#include namespace op { diff --git a/src/openpose/filestream/fileStream.cpp b/src/openpose/filestream/fileStream.cpp index 7bd81554..7bdc9f71 100644 --- a/src/openpose/filestream/fileStream.cpp +++ b/src/openpose/filestream/fileStream.cpp @@ -205,11 +205,12 @@ namespace op } } - void saveData(const std::vector& cvMats, const std::vector& cvMatNames, + void saveData(const std::vector& opMats, const std::vector& cvMatNames, const std::string& fileNameNoExtension, const DataFormat dataFormat) { try { + OP_OP2CVVECTORMAT(cvMats, opMats) // Sanity checks if (dataFormat == DataFormat::Json && CV_MAJOR_VERSION < 3) error(errorMessage, __LINE__, __FUNCTION__, __FILE__); @@ -229,12 +230,12 @@ namespace op } } - void saveData(const cv::Mat& cvMat, const std::string cvMatName, const std::string& fileNameNoExtension, + void saveData(const Matrix& opMat, const std::string cvMatName, const std::string& fileNameNoExtension, const DataFormat dataFormat) { try { - saveData(std::vector{cvMat}, std::vector{cvMatName}, fileNameNoExtension, + saveData(std::vector{opMat}, std::vector{cvMatName}, fileNameNoExtension, dataFormat); } catch (const std::exception& e) @@ -243,7 +244,7 @@ namespace op } } - std::vector loadData(const std::vector& cvMatNames, const std::string& fileNameNoExtension, + std::vector loadData(const std::vector& cvMatNames, const std::string& fileNameNoExtension, const DataFormat dataFormat) { try @@ -262,7 +263,8 @@ namespace op for (auto i = 0u ; i < cvMats.size() ; i++) fileStorage[cvMatNames[i]] >> cvMats[i]; fileStorage.release(); - return cvMats; + OP_CV2OPVECTORMAT(opMats, cvMats) + return opMats; } catch (const std::exception& e) { @@ -271,16 +273,16 @@ namespace op } } - cv::Mat loadData(const std::string& cvMatName, const std::string& fileNameNoExtension, const DataFormat dataFormat) + Matrix loadData(const std::string& cvMatName, const std::string& fileNameNoExtension, const DataFormat dataFormat) { try { - return loadData(std::vector{cvMatName}, fileNameNoExtension, dataFormat)[0]; + return OP_CV2OPMAT(loadData(std::vector{cvMatName}, fileNameNoExtension, dataFormat)[0]); } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return {}; + return Matrix(); } } @@ -341,11 +343,12 @@ namespace op } } - void saveImage(const cv::Mat& cvMat, const std::string& fullFilePath, + void saveImage(const Matrix& matrix, const std::string& fullFilePath, const std::vector& openCvCompressionParams) { try { + const cv::Mat cvMat = OP_OP2CVCONSTMAT(matrix); if (!cv::imwrite(fullFilePath, cvMat, openCvCompressionParams)) error("Image could not be saved on " + fullFilePath + ".", __LINE__, __FUNCTION__, __FILE__); } @@ -355,19 +358,19 @@ namespace op } } - cv::Mat loadImage(const std::string& fullFilePath, const int openCvFlags) + Matrix loadImage(const std::string& fullFilePath, const int openCvFlags) { try { cv::Mat cvMat = cv::imread(fullFilePath, openCvFlags); if (cvMat.empty()) log("Empty image on path: " + fullFilePath + ".", Priority::Max, __LINE__, __FUNCTION__, __FILE__); - return cvMat; + return OP_CV2OPMAT(cvMat); } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } diff --git a/src/openpose/filestream/heatMapSaver.cpp b/src/openpose/filestream/heatMapSaver.cpp index 632eb7e0..8db81136 100644 --- a/src/openpose/filestream/heatMapSaver.cpp +++ b/src/openpose/filestream/heatMapSaver.cpp @@ -53,7 +53,7 @@ namespace op else { // heatMaps -> cvOutputDatas - std::vector cvOutputDatas(heatMaps.size()); + std::vector cvOutputDatas(heatMaps.size()); for (auto i = 0u; i < cvOutputDatas.size(); i++) unrollArrayToUCharCvMat(cvOutputDatas[i], heatMaps[i]); // Save each heatMap diff --git a/src/openpose/filestream/imageSaver.cpp b/src/openpose/filestream/imageSaver.cpp index 72154185..ccc04a15 100644 --- a/src/openpose/filestream/imageSaver.cpp +++ b/src/openpose/filestream/imageSaver.cpp @@ -22,11 +22,11 @@ namespace op { } - void ImageSaver::saveImages(const cv::Mat& cvOutputData, const std::string& fileName) const + void ImageSaver::saveImages(const Matrix& cvOutputData, const std::string& fileName) const { try { - saveImages(std::vector{cvOutputData}, fileName); + saveImages(std::vector{cvOutputData}, fileName); } catch (const std::exception& e) { @@ -34,24 +34,24 @@ namespace op } } - void ImageSaver::saveImages(const std::vector& cvOutputDatas, const std::string& fileName) const + void ImageSaver::saveImages(const std::vector& matOutputDatas, const std::string& fileName) const { try { // Record cv::mat - if (!cvOutputDatas.empty()) + if (!matOutputDatas.empty()) { // File path (no extension) const auto fileNameNoExtension = getNextFileName(fileName) + "_rendered"; // Get names for each image - std::vector fileNames(cvOutputDatas.size()); + std::vector fileNames(matOutputDatas.size()); for (auto i = 0u; i < fileNames.size(); i++) fileNames[i] = {fileNameNoExtension + (i != 0 ? "_" + std::to_string(i) : "") + "." + mImageFormat}; // Save each image - for (auto i = 0u; i < cvOutputDatas.size(); i++) - saveImage(cvOutputDatas[i], fileNames[i]); + for (auto i = 0u; i < matOutputDatas.size(); i++) + saveImage(matOutputDatas[i], fileNames[i]); } } catch (const std::exception& e) diff --git a/src/openpose/filestream/keypointSaver.cpp b/src/openpose/filestream/keypointSaver.cpp index fef61e8b..b5247364 100644 --- a/src/openpose/filestream/keypointSaver.cpp +++ b/src/openpose/filestream/keypointSaver.cpp @@ -23,17 +23,17 @@ namespace op const auto fileNameNoExtension = getNextFileName(fileName) + "_" + keypointName; // Get vector of people poses - std::vector cvMatPoses(keypointVector.size()); + std::vector matPoses(keypointVector.size()); for (auto i = 0u; i < keypointVector.size(); i++) - cvMatPoses[i] = keypointVector[i].getConstCvMat(); + matPoses[i] = keypointVector[i].getConstCvMat(); // Get names inside file - std::vector keypointVectorNames(cvMatPoses.size()); - for (auto i = 0u; i < cvMatPoses.size(); i++) + std::vector keypointVectorNames(matPoses.size()); + for (auto i = 0u; i < matPoses.size(); i++) keypointVectorNames[i] = {keypointName + "_" + std::to_string(i)}; // Record people poses in desired format - saveData(cvMatPoses, keypointVectorNames, fileNameNoExtension, mFormat); + saveData(matPoses, keypointVectorNames, fileNameNoExtension, mFormat); } } catch (const std::exception& e) diff --git a/src/openpose/filestream/videoSaver.cpp b/src/openpose/filestream/videoSaver.cpp index 62eb2c5e..0ccbc4e0 100644 --- a/src/openpose/filestream/videoSaver.cpp +++ b/src/openpose/filestream/videoSaver.cpp @@ -143,7 +143,7 @@ namespace op auto codeAnswerAudio = system(audioCommand.c_str()); // Move temp output to real output if (codeAnswerAudio == 0) - codeAnswerAudio = system(("mv " + tempOutput + " " + upImpl->mVideoSaverPath).c_str()); + codeAnswerAudio = system(("mv " + tempOutput + " " + upImpl->mVideoSaverPath).c_str()); // Sanity check if (codeAnswerAudio != 0) log("\nVideo " + upImpl->mVideoSaverPath + " could not be saved with audio (exit code: " @@ -176,11 +176,11 @@ namespace op } } - void VideoSaver::write(const cv::Mat& cvMat) + void VideoSaver::write(const Matrix& matToSave) { try { - write(std::vector{cvMat}); + write(std::vector{matToSave}); } catch (const std::exception& e) { @@ -188,10 +188,11 @@ namespace op } } - void VideoSaver::write(const std::vector& cvMats) + void VideoSaver::write(const std::vector& matsToSave) { try { + OP_OP2CVVECTORMAT(cvMats, matsToSave); // Sanity check if (cvMats.empty()) error("The image(s) to be saved cannot be empty.", __LINE__, __FUNCTION__, __FILE__); @@ -236,7 +237,8 @@ namespace op // FFmpeg video if (upImpl->mUseFfmpeg) { - upImpl->upImageSaver->saveImages(cvOutputData, toFixedLengthString(upImpl->mImageSaverCounter, 12u)); + const auto opMat = OP_CV2OPMAT(cvOutputData); + upImpl->upImageSaver->saveImages(opMat, toFixedLengthString(upImpl->mImageSaverCounter, 12u)); upImpl->mImageSaverCounter++; } // OpenCV video diff --git a/src/openpose/gpu/cuda.cu b/src/openpose/gpu/cuda.cu index 91624620..c9e9c7dd 100644 --- a/src/openpose/gpu/cuda.cu +++ b/src/openpose/gpu/cuda.cu @@ -1,7 +1,7 @@ #ifdef USE_CUDA #include #include - #include + #include #endif #include diff --git a/src/openpose/gpu/gpu.cpp b/src/openpose/gpu/gpu.cpp index 91911855..ea75bccf 100644 --- a/src/openpose/gpu/gpu.cpp +++ b/src/openpose/gpu/gpu.cpp @@ -1,10 +1,10 @@ +#include #ifdef USE_CUDA #include #endif #ifdef USE_OPENCL - #include + #include #endif -#include namespace op { diff --git a/src/openpose/gpu/opencl.cpp b/src/openpose/gpu/opencl.cpp index 40273ac7..e0d38643 100644 --- a/src/openpose/gpu/opencl.cpp +++ b/src/openpose/gpu/opencl.cpp @@ -1,8 +1,8 @@ +#include // Must be before below includes #include #include -#include // Must be before below includes #ifdef USE_OPENCL - #include + #include #include #include #endif diff --git a/src/openpose/gui/frameDisplayer.cpp b/src/openpose/gui/frameDisplayer.cpp index dea9f9ac..9b8944bc 100644 --- a/src/openpose/gui/frameDisplayer.cpp +++ b/src/openpose/gui/frameDisplayer.cpp @@ -1,5 +1,5 @@ -#include // cv::imshow, cv::waitKey, cv::namedWindow, cv::setWindowProperty #include + #include namespace op { @@ -31,7 +31,8 @@ namespace op { setFullScreenMode(mFullScreenMode); - const cv::Mat blackFrame(mWindowedSize.y, mWindowedSize.x, CV_32FC3, {0,0,0}); + const cv::Mat cvBlackFrame(mWindowedSize.y, mWindowedSize.x, CV_32FC3, {0,0,0}); + const Matrix blackFrame = OP_CV2OPCONSTMAT(cvBlackFrame); FrameDisplayer::displayFrame(blackFrame); // This one will show most probably a white image (I guess the program does not have time to render // in 1 msec) @@ -90,24 +91,25 @@ namespace op } } - void FrameDisplayer::displayFrame(const cv::Mat& frame, const int waitKeyValue) + void FrameDisplayer::displayFrame(const Matrix& frame, const int waitKeyValue) { try { // Sanity check if (frame.empty()) error("Empty frame introduced.", __LINE__, __FUNCTION__, __FILE__); + const cv::Mat cvFrame = OP_OP2CVCONSTMAT(frame); // If frame > window size --> Resize window - if (mWindowedSize.x < frame.cols || mWindowedSize.y < frame.rows) + if (mWindowedSize.x < cvFrame.cols || mWindowedSize.y < cvFrame.rows) { - mWindowedSize.x = std::max(mWindowedSize.x, frame.cols); - mWindowedSize.y = std::max(mWindowedSize.y, frame.rows); + mWindowedSize.x = std::max(mWindowedSize.x, cvFrame.cols); + mWindowedSize.y = std::max(mWindowedSize.y, cvFrame.rows); cv::resizeWindow(mWindowName, mWindowedSize.x, mWindowedSize.y); // This one will show most probably a white image (I guess the program does not have time to render // in 1 msec) cv::waitKey(1); } - cv::imshow(mWindowName, frame); + cv::imshow(mWindowName, cvFrame); if (waitKeyValue != -1) cv::waitKey(waitKeyValue); } @@ -117,13 +119,13 @@ namespace op } } - void FrameDisplayer::displayFrame(const std::vector& frames, const int waitKeyValue) + void FrameDisplayer::displayFrame(const std::vector& frames, const int waitKeyValue) { try { // No frames if (frames.empty()) - displayFrame(cv::Mat(), waitKeyValue); + displayFrame(Matrix(), waitKeyValue); // 1 frame else if (frames.size() == 1u) displayFrame(frames[0], waitKeyValue); @@ -132,12 +134,17 @@ namespace op { // Prepare final cvMat // Concat (0) - cv::Mat cvMat = frames[0].clone(); + Matrix opMat = frames[0].clone(); + cv::Mat cvMat = OP_OP2CVMAT(opMat); // Concat (1,size()-1) for (auto i = 1u; i < frames.size(); i++) - cv::hconcat(cvMat, frames[i], cvMat); + { + const cv::Mat framesI = OP_OP2CVCONSTMAT(frames[i]); + cv::hconcat(cvMat, framesI, cvMat); + } + opMat = OP_CV2OPMAT(cvMat); // Display it - displayFrame(cvMat, waitKeyValue); + displayFrame(opMat, waitKeyValue); } } catch (const std::exception& e) diff --git a/src/openpose/gui/gui.cpp b/src/openpose/gui/gui.cpp index 72282d49..4a4b7621 100644 --- a/src/openpose/gui/gui.cpp +++ b/src/openpose/gui/gui.cpp @@ -15,7 +15,7 @@ namespace op { const auto fullScreen = false; FrameDisplayer frameDisplayer{OPEN_POSE_NAME_AND_VERSION + " - GUI Help", - Point{helpCvMat.cols, helpCvMat.rows}, fullScreen}; + Point{helpCvMat.cols(), helpCvMat.rows()}, fullScreen}; frameDisplayer.displayFrame(helpCvMat, 33); } } @@ -247,11 +247,11 @@ namespace op mFrameDisplayer.initializationOnThread(); } - void Gui::setImage(const cv::Mat& cvMatOutput) + void Gui::setImage(const Matrix& cvMatOutput) { try { - setImage(std::vector{cvMatOutput}); + setImage(std::vector{cvMatOutput}); } catch (const std::exception& e) { @@ -259,7 +259,7 @@ namespace op } } - void Gui::setImage(const std::vector& cvMatOutputs) + void Gui::setImage(const std::vector& cvMatOutputs) { try { diff --git a/src/openpose/gui/gui3D.cpp b/src/openpose/gui/gui3D.cpp index f5f4a3b9..f453fb6e 100644 --- a/src/openpose/gui/gui3D.cpp +++ b/src/openpose/gui/gui3D.cpp @@ -484,7 +484,7 @@ namespace op const Array& leftHandKeypoints3D, const Array& rightHandKeypoints3D) { try - { + { // 3-D rendering #ifdef USE_3D_RENDERER if (mDisplayMode == DisplayMode::DisplayAll || mDisplayMode == DisplayMode::Display3D) @@ -556,34 +556,35 @@ namespace op } } - cv::Mat Gui3D::readCvMat() + Matrix Gui3D::readCvMat() { try { // 3-D rendering - cv::Mat image; + cv::Mat cvImage; #ifdef USE_3D_RENDERER if (mDisplayMode == DisplayMode::DisplayAll || mDisplayMode == DisplayMode::Display3D) { // Save/display 3D display in OpenCV window if (mCopyGlToCvMat) { - image = cv::Mat(WINDOW_HEIGHT, WINDOW_WIDTH, CV_8UC3); + cvImage = cv::Mat(WINDOW_HEIGHT, WINDOW_WIDTH, CV_8UC3); #ifdef _WIN32 - glReadPixels(0, 0, WINDOW_WIDTH, WINDOW_HEIGHT, GL_BGR_EXT, GL_UNSIGNED_BYTE, image.data); + glReadPixels(0, 0, WINDOW_WIDTH, WINDOW_HEIGHT, GL_BGR_EXT, GL_UNSIGNED_BYTE, cvImage.data); #else - glReadPixels(0, 0, WINDOW_WIDTH, WINDOW_HEIGHT, GL_BGR, GL_UNSIGNED_BYTE, image.data); + glReadPixels(0, 0, WINDOW_WIDTH, WINDOW_HEIGHT, GL_BGR, GL_UNSIGNED_BYTE, cvImage.data); #endif - cv::flip(image, image, 0); + cv::flip(cvImage, cvImage, 0); } } #endif + Matrix image = OP_CV2OPMAT(cvImage); return image; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } } diff --git a/src/openpose/gui/guiAdam.cpp b/src/openpose/gui/guiAdam.cpp index b52eb5d4..62f6b025 100644 --- a/src/openpose/gui/guiAdam.cpp +++ b/src/openpose/gui/guiAdam.cpp @@ -275,7 +275,7 @@ namespace op void GuiAdam::update() { try - { + { // 2-D rendering if (mDisplayMode == DisplayMode::DisplayAll || mDisplayMode == DisplayMode::Display2D) Gui::update(); diff --git a/src/openpose/gui/guiInfoAdder.cpp b/src/openpose/gui/guiInfoAdder.cpp index 5abdb683..c0bc220e 100644 --- a/src/openpose/gui/guiInfoAdder.cpp +++ b/src/openpose/gui/guiInfoAdder.cpp @@ -1,8 +1,8 @@ +#include #include // std::snprintf #include // std::numeric_limits #include -#include -#include +#include namespace op { @@ -117,15 +117,16 @@ namespace op { } - void GuiInfoAdder::addInfo(cv::Mat& cvOutputData, const int numberPeople, const unsigned long long id, + void GuiInfoAdder::addInfo(Matrix& outputData, const int numberPeople, const unsigned long long id, const std::string& elementRenderedName, const unsigned long long frameNumber, const Array& poseIds, const Array& poseKeypoints) { try { + cv::Mat cvOutputData = OP_OP2CVMAT(outputData); // Sanity check if (cvOutputData.empty()) - error("Wrong input element (empty cvOutputData).", __LINE__, __FUNCTION__, __FILE__); + error("Wrong input element (empty outputData).", __LINE__, __FUNCTION__, __FILE__); // Size const auto borderMargin = positiveIntRound(fastMax(cvOutputData.cols, cvOutputData.rows) * 0.025); // Update fps diff --git a/src/openpose/hand/CMakeLists.txt b/src/openpose/hand/CMakeLists.txt index ab0c793b..c54476c7 100644 --- a/src/openpose/hand/CMakeLists.txt +++ b/src/openpose/hand/CMakeLists.txt @@ -23,7 +23,7 @@ if (UNIX OR APPLE) endif () target_link_libraries(openpose_hand openpose_core) - + if (BUILD_CAFFE) add_dependencies(openpose_hand openpose) endif (BUILD_CAFFE) diff --git a/src/openpose/hand/handDetector.cpp b/src/openpose/hand/handDetector.cpp index 34334e84..e091f5be 100644 --- a/src/openpose/hand/handDetector.cpp +++ b/src/openpose/hand/handDetector.cpp @@ -3,7 +3,7 @@ #include #include #include - + namespace op { inline Rectangle getHandFromPoseIndexes(const Array& poseKeypoints, const unsigned int person, const unsigned int wrist, diff --git a/src/openpose/hand/handDetectorFromTxt.cpp b/src/openpose/hand/handDetectorFromTxt.cpp index d0ee11a7..c2ae00e8 100644 --- a/src/openpose/hand/handDetectorFromTxt.cpp +++ b/src/openpose/hand/handDetectorFromTxt.cpp @@ -1,7 +1,7 @@ #include #include #include - + namespace op { std::vector getTxtPathsOnDirectory(const std::string& txtDirectoryPath) diff --git a/src/openpose/hand/handExtractorCaffe.cpp b/src/openpose/hand/handExtractorCaffe.cpp index 4a170316..c01c1364 100644 --- a/src/openpose/hand/handExtractorCaffe.cpp +++ b/src/openpose/hand/handExtractorCaffe.cpp @@ -1,7 +1,7 @@ +#include #ifdef USE_CAFFE #include #endif -#include // CV_WARP_INVERSE_MAP, CV_INTER_LINEAR #include #include #include @@ -10,14 +10,14 @@ #include #include #include -#include +#include namespace op { struct HandExtractorCaffe::ImplHandExtractorCaffe { #ifdef USE_CAFFE - bool netInitialized; + bool mNetInitialized; const int mGpuId; std::shared_ptr spNetCaffe; std::shared_ptr> spResizeAndMergeCaffe; @@ -29,7 +29,7 @@ namespace op ImplHandExtractorCaffe(const std::string& modelFolder, const int gpuId, const bool enableGoogleLogging) : - netInitialized{false}, + mNetInitialized{false}, mGpuId{gpuId}, spNetCaffe{std::make_shared(modelFolder + HAND_PROTOTXT, modelFolder + HAND_TRAINED_MODEL, gpuId, enableGoogleLogging)}, @@ -65,7 +65,7 @@ namespace op CV_INTER_LINEAR | CV_WARP_INVERSE_MAP, cv::BORDER_CONSTANT, cv::Scalar{0,0,0}); // CV_INTER_CUBIC | CV_WARP_INVERSE_MAP, cv::BORDER_CONSTANT, cv::Scalar{0,0,0}); // cv::Mat -> float* - uCharCvMatToFloatPtr(handImageCrop.getPtr(), handImage, true); + uCharCvMatToFloatPtr(handImageCrop.getPtr(), OP_CV2OPMAT(handImage), true); } catch (const std::exception& e) { @@ -187,11 +187,57 @@ namespace op error(e.what(), __LINE__, __FUNCTION__, __FILE__); } } + + void detectHandKeypoints( + Array& handCurrent, std::shared_ptr& netCaffe, std::shared_ptr>& resizeAndMergeCaffe, + std::shared_ptr>& maximumCaffe, std::shared_ptr>& caffeNetOutputBlob, + std::shared_ptr>& heatMapsBlob, std::shared_ptr>& peaksBlob, bool& netInitialized, + Array& handImageCrop, const int person, const cv::Mat& affineMatrix, const int gpuId) + { + try + { + #ifdef USE_CAFFE + // 1. Deep net + netCaffe->forwardPass(handImageCrop); + + // Reshape blobs + if (!netInitialized) + { + netInitialized = true; + reshapeHandExtractorCaffe( + resizeAndMergeCaffe, maximumCaffe, caffeNetOutputBlob, heatMapsBlob, peaksBlob, gpuId); + } + + // 2. Resize heat maps + merge different scales + resizeAndMergeCaffe->Forward({caffeNetOutputBlob.get()}, {heatMapsBlob.get()}); + + // 3. Get peaks by Non-Maximum Suppression + maximumCaffe->Forward({heatMapsBlob.get()}, {peaksBlob.get()}); + + // Estimate keypoint locations + connectKeypoints( + handCurrent, person, affineMatrix, peaksBlob->mutable_cpu_data()); + + // 5. CUDA sanity check + #ifdef USE_CUDA + cudaCheck(__LINE__, __FUNCTION__, __FILE__); + #endif + #else + UNUSED(handCurrent); + UNUSED(person); + UNUSED(affineMatrix); + #endif + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } #endif HandExtractorCaffe::HandExtractorCaffe(const Point& netInputSize, const Point& netOutputSize, const std::string& modelFolder, const int gpuId, - const unsigned short numberScales, + const int numberScales, const float rangeScales, const std::vector& heatMapTypes, const ScaleMode heatMapScaleMode, const bool enableGoogleLogging) : @@ -256,13 +302,15 @@ namespace op } void HandExtractorCaffe::forwardPass( - const std::vector, 2>> handRectangles, const cv::Mat& cvInputData) + const std::vector, 2>> handRectangles, const Matrix& inputData) { try { #ifdef USE_CAFFE if (mEnabled && !handRectangles.empty()) { + const cv::Mat cvInputData = OP_OP2CVCONSTMAT(inputData); + // Sanity check if (cvInputData.empty()) error("Empty cvInputData.", __LINE__, __FUNCTION__, __FILE__); @@ -330,7 +378,11 @@ namespace op cropFrame(mHandImageCrop, affineMatrix, cvInputData, handRectangle, netInputSide, mNetOutputSize, mirrorImage); // Deep net + Estimate keypoint locations - detectHandKeypoints(handCurrent, person, affineMatrix); + detectHandKeypoints( + handCurrent, upImpl->spNetCaffe, upImpl->spResizeAndMergeCaffe, + upImpl->spMaximumCaffe, upImpl->spCaffeNetOutputBlob, + upImpl->spHeatMapsBlob, upImpl->spPeaksBlob, upImpl->mNetInitialized, + mHandImageCrop, person, affineMatrix, upImpl->mGpuId); } // Multi-scale detection else @@ -367,7 +419,11 @@ namespace op cropFrame(mHandImageCrop, affineMatrix, cvInputData, handRectangleScale, netInputSide, mNetOutputSize, mirrorImage); // Deep net + Estimate keypoint locations - detectHandKeypoints(handEstimated, 0, affineMatrix); + detectHandKeypoints( + handEstimated, upImpl->spNetCaffe, upImpl->spResizeAndMergeCaffe, + upImpl->spMaximumCaffe, upImpl->spCaffeNetOutputBlob, + upImpl->spHeatMapsBlob, upImpl->spPeaksBlob, upImpl->mNetInitialized, + mHandImageCrop, 0, affineMatrix, upImpl->mGpuId); if (i == 0 || getAverageScore(handEstimated,0) > getAverageScore(handCurrent,person)) std::copy(handEstimated.getConstPtr(), @@ -405,49 +461,4 @@ namespace op error(e.what(), __LINE__, __FUNCTION__, __FILE__); } } - - void HandExtractorCaffe::detectHandKeypoints(Array& handCurrent, const int person, - const cv::Mat& affineMatrix) - { - try - { - #ifdef USE_CAFFE - // 1. Deep net - upImpl->spNetCaffe->forwardPass(mHandImageCrop); - - // Reshape blobs - if (!upImpl->netInitialized) - { - upImpl->netInitialized = true; - reshapeHandExtractorCaffe(upImpl->spResizeAndMergeCaffe, upImpl->spMaximumCaffe, - upImpl->spCaffeNetOutputBlob, upImpl->spHeatMapsBlob, - upImpl->spPeaksBlob, upImpl->mGpuId); - } - - // 2. Resize heat maps + merge different scales - upImpl->spResizeAndMergeCaffe->Forward( - {upImpl->spCaffeNetOutputBlob.get()}, {upImpl->spHeatMapsBlob.get()}); - - // 3. Get peaks by Non-Maximum Suppression - upImpl->spMaximumCaffe->Forward({upImpl->spHeatMapsBlob.get()}, {upImpl->spPeaksBlob.get()}); - - // Estimate keypoint locations - connectKeypoints(handCurrent, person, affineMatrix, - upImpl->spPeaksBlob->mutable_cpu_data()); - - // 5. CUDA sanity check - #ifdef USE_CUDA - cudaCheck(__LINE__, __FUNCTION__, __FILE__); - #endif - #else - UNUSED(handCurrent); - UNUSED(person); - UNUSED(affineMatrix); - #endif - } - catch (const std::exception& e) - { - error(e.what(), __LINE__, __FUNCTION__, __FILE__); - } - } } diff --git a/src/openpose/hand/handExtractorNet.cpp b/src/openpose/hand/handExtractorNet.cpp index 392a6216..8793c927 100644 --- a/src/openpose/hand/handExtractorNet.cpp +++ b/src/openpose/hand/handExtractorNet.cpp @@ -4,7 +4,7 @@ namespace op { HandExtractorNet::HandExtractorNet(const Point& netInputSize, const Point& netOutputSize, - const unsigned short numberScales, const float rangeScales, + const int numberScales, const float rangeScales, const std::vector& heatMapTypes, const ScaleMode heatMapScaleMode) : mMultiScaleNumberAndRange{std::make_pair(numberScales, rangeScales)}, diff --git a/src/openpose/hand/renderHand.cu b/src/openpose/hand/renderHand.cu index d3f478db..b0715d91 100644 --- a/src/openpose/hand/renderHand.cu +++ b/src/openpose/hand/renderHand.cu @@ -1,8 +1,8 @@ -#include -#include -#include -#include #include +#include +#include +#include +#include namespace op { diff --git a/src/openpose/net/CMakeLists.txt b/src/openpose/net/CMakeLists.txt index 22c8e859..385e95f1 100644 --- a/src/openpose/net/CMakeLists.txt +++ b/src/openpose/net/CMakeLists.txt @@ -31,7 +31,7 @@ if (UNIX OR APPLE) endif () add_library(caffe SHARED IMPORTED) - set_property(TARGET caffe PROPERTY IMPORTED_LOCATION ${Caffe_LIBS}) + set_property(TARGET caffe PROPERTY IMPORTED_LOCATION ${Caffe_LIBS}) target_link_libraries(openpose_net caffe ${MKL_LIBS} openpose_core) if (BUILD_CAFFE) diff --git a/src/openpose/net/bodyPartConnectorBase.cpp b/src/openpose/net/bodyPartConnectorBase.cpp index fc520175..844687e5 100644 --- a/src/openpose/net/bodyPartConnectorBase.cpp +++ b/src/openpose/net/bodyPartConnectorBase.cpp @@ -1,9 +1,11 @@ +#include +#include // std::sort +#include // std::sqrt #include #include #include #include #include -#include namespace op { @@ -385,7 +387,7 @@ namespace op rowVector[bodyPartPairs[1]] = indexB; rowVector.back() = 2; // add the score of parts and the connection - const auto personScore = peaksPtr[indexA] + peaksPtr[indexB] + score; + const auto personScore = T(peaksPtr[indexA] + peaksPtr[indexB] + score); peopleVector.emplace_back(std::make_pair(rowVector, personScore)); } } @@ -452,7 +454,7 @@ namespace op rowVector[bodyPartA] = indexA; rowVector[bodyPartB] = indexB; rowVector.back() = 2; - const auto personScore = peaksPtr[indexA] + peaksPtr[indexB] + score; + const auto personScore = T(peaksPtr[indexA] + peaksPtr[indexB] + score); peopleVector.emplace_back(std::make_pair(rowVector, personScore)); } } @@ -603,7 +605,7 @@ namespace op // Number keypoints rowVector.back() = 2; // Score - const auto personScore = peaksPtr[indexScoreA] + peaksPtr[indexScoreB] + pafScore; + const auto personScore = T(peaksPtr[indexScoreA] + peaksPtr[indexScoreB] + pafScore); // Set associated personAssigned as assigned aAssigned = (int)peopleVector.size(); bAssigned = aAssigned; diff --git a/src/openpose/net/bodyPartConnectorBaseCL.cpp b/src/openpose/net/bodyPartConnectorBaseCL.cpp index cee53a1a..3dbfa71c 100644 --- a/src/openpose/net/bodyPartConnectorBaseCL.cpp +++ b/src/openpose/net/bodyPartConnectorBaseCL.cpp @@ -1,12 +1,12 @@ -#ifdef USE_OPENCL - #include - #include -#endif +#include +#include #include #include #include -#include -#include +#ifdef USE_OPENCL + #include + #include +#endif namespace op { diff --git a/src/openpose/net/bodyPartConnectorCaffe.cpp b/src/openpose/net/bodyPartConnectorCaffe.cpp index 9ebdd208..ba66e0e7 100644 --- a/src/openpose/net/bodyPartConnectorCaffe.cpp +++ b/src/openpose/net/bodyPartConnectorCaffe.cpp @@ -1,17 +1,17 @@ +#include #ifdef USE_CAFFE #include #endif #ifdef USE_CUDA #include - #include -#endif -#ifdef USE_OPENCL - #include - #include + #include #endif #include #include -#include +#ifdef USE_OPENCL + #include + #include +#endif namespace op { diff --git a/src/openpose/net/maximumBase.cpp b/src/openpose/net/maximumBase.cpp index 31fff9a2..6404ae47 100644 --- a/src/openpose/net/maximumBase.cpp +++ b/src/openpose/net/maximumBase.cpp @@ -1,5 +1,6 @@ -// #include #include +// #include +#include // cv::Mat namespace op { diff --git a/src/openpose/net/netCaffe.cpp b/src/openpose/net/netCaffe.cpp index 62eba9eb..e8505154 100644 --- a/src/openpose/net/netCaffe.cpp +++ b/src/openpose/net/netCaffe.cpp @@ -1,3 +1,4 @@ +#include #include // std::accumulate #ifdef USE_CAFFE #include @@ -8,13 +9,12 @@ #ifdef USE_CUDA #include #endif -#ifdef USE_OPENCL - #include - #include -#endif #include #include -#include +#ifdef USE_OPENCL + #include + #include +#endif namespace op { diff --git a/src/openpose/net/netOpenCv.cpp b/src/openpose/net/netOpenCv.cpp index 63edb9da..ac48be82 100644 --- a/src/openpose/net/netOpenCv.cpp +++ b/src/openpose/net/netOpenCv.cpp @@ -1,11 +1,12 @@ // TODO: After completely adding the OpenCV DNN module, add this flag to CMake as alternative to USE_CAFFE // #define USE_OPEN_CV_DNN +#include // Note: OpenCV only uses CPU or OpenCL (for Intel GPUs). Used CUDA for following blobs (Resize + NMS) -#include // OPEN_CV_IS_4_OR_HIGHER #ifdef USE_CAFFE #include #endif +#include // OPEN_CV_IS_4_OR_HIGHER #ifdef USE_OPEN_CV_DNN #if defined(USE_CAFFE) && defined(USE_CUDA) && defined(OPEN_CV_IS_4_OR_HIGHER) #include @@ -17,7 +18,6 @@ #endif #include // std::accumulate #include -#include namespace op { diff --git a/src/openpose/net/nmsBase.cu b/src/openpose/net/nmsBase.cu index 6e008d82..ad27634f 100644 --- a/src/openpose/net/nmsBase.cu +++ b/src/openpose/net/nmsBase.cu @@ -1,8 +1,8 @@ +#include #include #include #include -#include -#include +#include namespace op { diff --git a/src/openpose/net/nmsBaseCL.cpp b/src/openpose/net/nmsBaseCL.cpp index 75877e67..a5f9d4d8 100644 --- a/src/openpose/net/nmsBaseCL.cpp +++ b/src/openpose/net/nmsBaseCL.cpp @@ -1,13 +1,13 @@ +#include #include #include #include #include -#ifdef USE_OPENCL - #include - #include -#endif #include -#include +#ifdef USE_OPENCL + #include + #include +#endif namespace op { @@ -135,7 +135,7 @@ namespace op { int x = get_global_id(0); int y = get_global_id(1); - int index = y*w + x; + int index = y*w + x; if (0 < x && x < (w-1) && 0 < y && y < (h-1)) { diff --git a/src/openpose/net/nmsCaffe.cpp b/src/openpose/net/nmsCaffe.cpp index bc3ebd04..e45a2e2a 100644 --- a/src/openpose/net/nmsCaffe.cpp +++ b/src/openpose/net/nmsCaffe.cpp @@ -1,12 +1,12 @@ +#include #ifdef USE_CAFFE #include #endif -#ifdef USE_OPENCL - #include - #include -#endif #include -#include +#ifdef USE_OPENCL + #include + #include +#endif namespace op { diff --git a/src/openpose/net/resizeAndMergeBase.cpp b/src/openpose/net/resizeAndMergeBase.cpp index a84071d9..155fa59b 100644 --- a/src/openpose/net/resizeAndMergeBase.cpp +++ b/src/openpose/net/resizeAndMergeBase.cpp @@ -1,8 +1,7 @@ -#include -#include +#include #include #include -#include +#include namespace op { diff --git a/src/openpose/net/resizeAndMergeBase.cu b/src/openpose/net/resizeAndMergeBase.cu index 5fba3af4..46fff6a3 100644 --- a/src/openpose/net/resizeAndMergeBase.cu +++ b/src/openpose/net/resizeAndMergeBase.cu @@ -1,6 +1,6 @@ -#include -#include #include +#include +#include namespace op { diff --git a/src/openpose/net/resizeAndMergeBaseCL.cpp b/src/openpose/net/resizeAndMergeBaseCL.cpp index 1c21f045..2516f0fc 100644 --- a/src/openpose/net/resizeAndMergeBaseCL.cpp +++ b/src/openpose/net/resizeAndMergeBaseCL.cpp @@ -1,10 +1,10 @@ -#ifdef USE_OPENCL - #include - #include -#endif -#include #include #include +#include +#ifdef USE_OPENCL + #include + #include +#endif namespace op { diff --git a/src/openpose/net/resizeAndMergeCaffe.cpp b/src/openpose/net/resizeAndMergeCaffe.cpp index 192cb087..0de40a14 100644 --- a/src/openpose/net/resizeAndMergeCaffe.cpp +++ b/src/openpose/net/resizeAndMergeCaffe.cpp @@ -1,12 +1,12 @@ +#include #ifdef USE_CAFFE #include #endif -#ifdef USE_OPENCL - #include - #include -#endif #include -#include +#ifdef USE_OPENCL + #include + #include +#endif namespace op { diff --git a/src/openpose/pose/CMakeLists.txt b/src/openpose/pose/CMakeLists.txt index 92b73c7c..4c84d8d3 100644 --- a/src/openpose/pose/CMakeLists.txt +++ b/src/openpose/pose/CMakeLists.txt @@ -24,7 +24,7 @@ if (UNIX OR APPLE) endif () target_link_libraries(openpose_pose openpose_core) - + if (BUILD_CAFFE) add_dependencies(openpose_pose openpose) endif (BUILD_CAFFE) diff --git a/src/openpose/pose/poseExtractor.cpp b/src/openpose/pose/poseExtractor.cpp index 7f4af396..ea8f6b49 100644 --- a/src/openpose/pose/poseExtractor.cpp +++ b/src/openpose/pose/poseExtractor.cpp @@ -133,7 +133,7 @@ namespace op } } - Array PoseExtractor::extractIds(const Array& poseKeypoints, const cv::Mat& cvMatInput, + Array PoseExtractor::extractIds(const Array& poseKeypoints, const Matrix& cvMatInput, const unsigned long long imageViewIndex) { try @@ -151,7 +151,7 @@ namespace op } Array PoseExtractor::extractIdsLockThread(const Array& poseKeypoints, - const cv::Mat& cvMatInput, + const Matrix& cvMatInput, const unsigned long long imageViewIndex, const long long frameId) { @@ -170,7 +170,7 @@ namespace op } void PoseExtractor::track(Array& poseKeypoints, Array& poseIds, - const cv::Mat& cvMatInput, + const Matrix& cvMatInput, const unsigned long long imageViewIndex) { try @@ -201,7 +201,7 @@ namespace op } void PoseExtractor::trackLockThread(Array& poseKeypoints, Array& poseIds, - const cv::Mat& cvMatInput, + const Matrix& cvMatInput, const unsigned long long imageViewIndex, const long long frameId) { try diff --git a/src/openpose/pose/poseExtractorCaffe.cpp b/src/openpose/pose/poseExtractorCaffe.cpp index dc3dff7b..d5e7c055 100644 --- a/src/openpose/pose/poseExtractorCaffe.cpp +++ b/src/openpose/pose/poseExtractorCaffe.cpp @@ -1,15 +1,13 @@ +#include #include // std::numeric_limits #include -#ifdef USE_CUDA - #include -#endif #include #include #include #include #include #include -#include +#include namespace op { @@ -293,7 +291,14 @@ namespace op // 2. Resize heat maps + merge different scales // ~5ms (GPU) / ~20ms (CPU) const auto caffeNetOutputBlobs = arraySharedToPtr(spCaffeNetOutputBlobs); - const std::vector floatScaleRatios(scaleInputToNetInputs.begin(), scaleInputToNetInputs.end()); + // Set and fill floatScaleRatios + // Option 1/2 (warning for double-to-float conversion) + // const std::vector floatScaleRatios(scaleInputToNetInputs.begin(), scaleInputToNetInputs.end()); + // Option 2/2 + std::vector floatScaleRatios; + std::for_each( + scaleInputToNetInputs.begin(), scaleInputToNetInputs.end(), + [&floatScaleRatios](const double value) { floatScaleRatios.emplace_back(float(value)); }); spResizeAndMergeCaffe->setScaleRatios(floatScaleRatios); spResizeAndMergeCaffe->Forward(caffeNetOutputBlobs, {spHeatMapsBlob.get()}); // Get scale net to output (i.e., image input) @@ -341,22 +346,22 @@ namespace op const auto rectangleF = getKeypointsRectangle(mPoseKeypoints, person, nmsThreshold) / mScaleNetToOutput; // Make rectangle bigger to make sure the whole body is inside - cv::Rect cvRectangle{ + Rectangle rectangleInt{ positiveIntRound(rectangleF.x - 0.2*rectangleF.width), positiveIntRound(rectangleF.y - 0.2*rectangleF.height), positiveIntRound(rectangleF.width*1.4), positiveIntRound(rectangleF.height*1.4) }; - keepRoiInside(cvRectangle, inputNetData[0].getSize(3), inputNetData[0].getSize(2)); + keepRoiInside(rectangleInt, inputNetData[0].getSize(3), inputNetData[0].getSize(2)); // Input size // // Note: In order to preserve speed but maximize accuracy // // If e.g. rectange = 10x1 and inputSize = 656x368 --> targetSize = 656x368 // // Note: If e.g. rectange = 1x10 and inputSize = 656x368 --> targetSize = 368x656 - // const auto width = ( ? cvRectangle.width : cvRectangle.height); - // const auto height = (width == cvRectangle.width ? cvRectangle.height : cvRectangle.width); + // const auto width = ( ? rectangleInt.width : rectangleInt.height); + // const auto height = (width == rectangleInt.width ? rectangleInt.height : rectangleInt.width); // const Point inputSize{width, height}; // Note: If inputNetData.size = -1x368 --> TargetSize = 368x-1 - const Point inputSizeInit{cvRectangle.width, cvRectangle.height}; + const Point inputSizeInit{rectangleInt.width, rectangleInt.height}; // Target size Point targetSize; // Optimal case (using training size) @@ -370,7 +375,7 @@ namespace op const auto maxSide = fastMin( 368, fastMax(inputNetData[0].getSize(2), inputNetData[0].getSize(3))); // Person bounding box is vertical - if (cvRectangle.width < cvRectangle.height) + if (rectangleInt.width < rectangleInt.height) targetSize = Point{minSide, maxSide}; // Person bounding box is horizontal else @@ -388,17 +393,17 @@ namespace op { if (padding.x > 2) // 2 pixels as threshold { - cvRectangle.x -= padding.x/2; - cvRectangle.width += padding.x; + rectangleInt.x -= padding.x/2; + rectangleInt.width += padding.x; } else if (padding.y > 2) // 2 pixels as threshold { - cvRectangle.y -= padding.y/2; - cvRectangle.height += padding.y; + rectangleInt.y -= padding.y/2; + rectangleInt.height += padding.y; } - keepRoiInside(cvRectangle, inputNetData[0].getSize(3), inputNetData[0].getSize(2)); + keepRoiInside(rectangleInt, inputNetData[0].getSize(3), inputNetData[0].getSize(2)); scaleNetToRoi = resizeGetScaleFactor( - Point{cvRectangle.width, cvRectangle.height}, targetSize); + Point{rectangleInt.width, rectangleInt.height}, targetSize); } // No if scaleNetToRoi < 1 (image would be shrinked, so we assume best result already obtained) if (scaleNetToRoi > 1) @@ -413,7 +418,8 @@ namespace op inputNetData[0].getSize(2), inputNetData[0].getSize(3), CV_32FC1, inputNetData[0].getPseudoConstPtr() + c * areaInput); // Input image cropped - const cv::Mat inputCvMat(wholeInputCvMat, cvRectangle); + const cv::Mat inputCvMat( + wholeInputCvMat, cv::Rect{rectangleInt.x, rectangleInt.y, rectangleInt.width, rectangleInt.height}); // Resize image for inputNetDataRoi cv::Mat resizedImageCvMat( inputNetDataRoi.getSize(2), inputNetDataRoi.getSize(3), CV_32FC1, @@ -445,7 +451,7 @@ namespace op const std::vector floatScaleRatiosNew{(float)scaleInputToNetInputs[0]}; spResizeAndMergeCaffe->setScaleRatios(floatScaleRatiosNew); spResizeAndMergeCaffe->Forward( - caffeNetOutputBlobsNew, {spHeatMapsBlob.get()}); + caffeNetOutputBlobsNew, {spHeatMapsBlob.get()}); // Get scale net to output (i.e., image input) const auto scaleRoiToOutput = float(mScaleNetToOutput / scaleNetToRoi); // 3. Get peaks by Non-Maximum Suppression @@ -467,8 +473,8 @@ namespace op if (!poseKeypoints.empty()) { // // Scale back keypoints - const auto xOffset = float(cvRectangle.x*mScaleNetToOutput); - const auto yOffset = float(cvRectangle.y*mScaleNetToOutput); + const auto xOffset = float(rectangleInt.x*mScaleNetToOutput); + const auto yOffset = float(rectangleInt.y*mScaleNetToOutput); scaleKeypoints2d(poseKeypoints, 1.f, 1.f, xOffset, yOffset); // Re-assign person back // // Option a) Just use biggest person (simplest but fails with crowded people) diff --git a/src/openpose/pose/poseExtractorNet.cpp b/src/openpose/pose/poseExtractorNet.cpp index 98194970..033f38ef 100644 --- a/src/openpose/pose/poseExtractorNet.cpp +++ b/src/openpose/pose/poseExtractorNet.cpp @@ -1,10 +1,11 @@ +#include +#include // std::round #ifdef USE_CUDA #include #include #endif #include #include -#include namespace op { diff --git a/src/openpose/pose/renderPose.cu b/src/openpose/pose/renderPose.cu index adaae213..2690f101 100644 --- a/src/openpose/pose/renderPose.cu +++ b/src/openpose/pose/renderPose.cu @@ -1,8 +1,8 @@ -#include -#include -#include -#include #include +#include +#include +#include +#include namespace op { diff --git a/src/openpose/producer/CMakeLists.txt b/src/openpose/producer/CMakeLists.txt index de1ff329..1187237a 100644 --- a/src/openpose/producer/CMakeLists.txt +++ b/src/openpose/producer/CMakeLists.txt @@ -1,4 +1,5 @@ set(SOURCES_OP_PRODUCER + datumProducer.cpp defineTemplates.cpp flirReader.cpp imageDirectoryReader.cpp @@ -16,7 +17,7 @@ set(SOURCES_OPENPOSE ${SOURCES_OPENPOSE} ${SOURCES_OP_PRODUCER_WITH_CP} PARENT_S if (UNIX OR APPLE) add_library(openpose_producer ${SOURCES_OP_PRODUCER}) - target_link_libraries(openpose_producer ${OpenCV_LIBS} openpose_core + target_link_libraries(openpose_producer ${OpenCV_LIBS} openpose_core openpose_thread openpose_filestream) install(TARGETS openpose_producer diff --git a/src/openpose/producer/datumProducer.cpp b/src/openpose/producer/datumProducer.cpp new file mode 100644 index 00000000..3944703e --- /dev/null +++ b/src/openpose/producer/datumProducer.cpp @@ -0,0 +1,145 @@ +#include +#include + +namespace op +{ + void datumProducerConstructor( + const std::shared_ptr& producerSharedPtr, + const unsigned long long frameFirst, const unsigned long long frameStep, const unsigned long long frameLast) + { + try + { + // Sanity check + if (frameLast < frameFirst) + error("The desired initial frame must be lower than the last one (flags `--frame_first` vs." + " `--frame_last`). Current: " + std::to_string(frameFirst) + " vs. " + std::to_string(frameLast) + + ".", __LINE__, __FUNCTION__, __FILE__); + if (frameLast != std::numeric_limits::max() + && frameLast > producerSharedPtr->get(getCvCapPropFrameCount())-1) + error("The desired last frame must be lower than the length of the video or the number of images." + " Current: " + std::to_string(frameLast) + " vs. " + + std::to_string(positiveIntRound(producerSharedPtr->get(getCvCapPropFrameCount()))-1) + ".", + __LINE__, __FUNCTION__, __FILE__); + // Set frame first and step + if (producerSharedPtr->getType() != ProducerType::FlirCamera + && producerSharedPtr->getType() != ProducerType::IPCamera + && producerSharedPtr->getType() != ProducerType::Webcam) + { + // Frame first + producerSharedPtr->set(CV_CAP_PROP_POS_FRAMES, (double)frameFirst); + // Frame step + producerSharedPtr->set(ProducerProperty::FrameStep, (double)frameStep); + } + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } + + void datumProducerConstructorTooManyConsecutiveEmptyFrames( + unsigned int& numberConsecutiveEmptyFrames, const bool emptyFrame) + { + try + { + numberConsecutiveEmptyFrames = (emptyFrame ? numberConsecutiveEmptyFrames+1 : 0); + const auto threshold = 500u; + if (numberConsecutiveEmptyFrames >= threshold) + error("Detected too many (" + std::to_string(numberConsecutiveEmptyFrames) + + ") empty frames in a row.", __LINE__, __FUNCTION__, __FILE__); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } + + bool datumProducerConstructorRunningAndGetDatumIsDatumProducerRunning( + const std::shared_ptr& producerSharedPtr, const unsigned long long numberFramesToProcess, + const unsigned long long globalCounter) + { + try + { + // Check last desired frame has not been reached + if (numberFramesToProcess != std::numeric_limits::max() + && globalCounter > numberFramesToProcess) + { + producerSharedPtr->release(); + } + // If producer released -> it sends an empty Mat + a datumProducerRunning signal + return producerSharedPtr->isOpened(); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return false; + } + } + + void datumProducerConstructorRunningAndGetDatumApplyPlayerControls( + const std::shared_ptr& producerSharedPtr, + const std::shared_ptr, std::atomic>>& videoSeekSharedPtr) + { + try + { + // Fast forward/backward - Seek to specific frame index desired + if (videoSeekSharedPtr != nullptr) + { + // Fake pause vs. normal mode + const auto increment = videoSeekSharedPtr->second - (videoSeekSharedPtr->first ? 1 : 0); + // Normal mode + if (increment != 0) + producerSharedPtr->set( + CV_CAP_PROP_POS_FRAMES, producerSharedPtr->get(CV_CAP_PROP_POS_FRAMES) + increment); + // It must be always reset or bug in fake pause + videoSeekSharedPtr->second = 0; + } + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } + + unsigned long long datumProducerConstructorRunningAndGetNextFrameNumber( + const std::shared_ptr& producerSharedPtr) + { + try + { + // Get next frame number + return (unsigned long long)producerSharedPtr->get(CV_CAP_PROP_POS_FRAMES); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return 0ull; + } + } + + void datumProducerConstructorRunningAndGetDatumFrameIntegrity(Matrix& inputDataMatrix) + { + try + { + // Image integrity + if (inputDataMatrix.channels() != 3) + { + const std::string commonMessage{"Input images must be 3-channel BGR."}; + // Grey to RGB if required + if (inputDataMatrix.channels() == 1) + { + log(commonMessage + " Converting grey image into BGR.", Priority::High); + cv::Mat inputData = OP_OP2CVMAT(inputDataMatrix); + cv::cvtColor(inputData, inputData, CV_GRAY2BGR); + // Diferent memory size --> new cv::Mat raw ptr memory --> new Matrix + inputDataMatrix = OP_CV2OPMAT(inputData); + } + else + error(commonMessage, __LINE__, __FUNCTION__, __FILE__); + } + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } +} diff --git a/src/openpose/producer/flirReader.cpp b/src/openpose/producer/flirReader.cpp index e3500be4..c1b26f5c 100644 --- a/src/openpose/producer/flirReader.cpp +++ b/src/openpose/producer/flirReader.cpp @@ -1,6 +1,7 @@ +#include #include #include -#include +#include namespace op { @@ -36,7 +37,7 @@ namespace op } } - std::vector FlirReader::getCameraMatrices() + std::vector FlirReader::getCameraMatrices() { try { @@ -49,7 +50,7 @@ namespace op } } - std::vector FlirReader::getCameraExtrinsics() + std::vector FlirReader::getCameraExtrinsics() { try { @@ -62,7 +63,7 @@ namespace op } } - std::vector FlirReader::getCameraIntrinsics() + std::vector FlirReader::getCameraIntrinsics() { try { @@ -114,7 +115,7 @@ namespace op } } - cv::Mat FlirReader::getRawFrame() + Matrix FlirReader::getRawFrame() { try { @@ -123,11 +124,11 @@ namespace op catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } - std::vector FlirReader::getRawFrames() + std::vector FlirReader::getRawFrames() { try { diff --git a/src/openpose/producer/imageDirectoryReader.cpp b/src/openpose/producer/imageDirectoryReader.cpp index 1a0eaeb0..b3086282 100644 --- a/src/openpose/producer/imageDirectoryReader.cpp +++ b/src/openpose/producer/imageDirectoryReader.cpp @@ -1,7 +1,8 @@ +#include #include #include #include -#include +#include namespace op { @@ -52,7 +53,7 @@ namespace op } } - cv::Mat ImageDirectoryReader::getRawFrame() + Matrix ImageDirectoryReader::getRawFrame() { try { @@ -66,22 +67,22 @@ namespace op // after setWidth/setHeight this is performed over the new resolution (so they always match). checkFrameIntegrity(frame); // Update size, since images might have different size between each one of them - mResolution = Point{frame.cols, frame.rows}; + mResolution = Point{frame.cols(), frame.rows()}; // Return final frame return frame; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } - std::vector ImageDirectoryReader::getRawFrames() + std::vector ImageDirectoryReader::getRawFrames() { try { - std::vector rawFrames; + std::vector rawFrames; for (auto i = 0 ; i < positiveIntRound(Producer::get(ProducerProperty::NumberViews)) ; i++) rawFrames.emplace_back(getRawFrame()); return rawFrames; diff --git a/src/openpose/producer/ipCameraReader.cpp b/src/openpose/producer/ipCameraReader.cpp index 129f9598..b75c1b58 100644 --- a/src/openpose/producer/ipCameraReader.cpp +++ b/src/openpose/producer/ipCameraReader.cpp @@ -30,7 +30,7 @@ namespace op } } - cv::Mat IpCameraReader::getRawFrame() + Matrix IpCameraReader::getRawFrame() { try { @@ -39,11 +39,11 @@ namespace op catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } - std::vector IpCameraReader::getRawFrames() + std::vector IpCameraReader::getRawFrames() { try { diff --git a/src/openpose/producer/producer.cpp b/src/openpose/producer/producer.cpp index 00849a99..314b688e 100644 --- a/src/openpose/producer/producer.cpp +++ b/src/openpose/producer/producer.cpp @@ -1,9 +1,10 @@ +#include #include #include #include #include #include -#include +#include namespace op { @@ -83,26 +84,26 @@ namespace op Producer::~Producer(){} - cv::Mat Producer::getFrame() + Matrix Producer::getFrame() { try { // Return first element from getFrames (if any) const auto frames = getFrames(); - return (frames.empty() ? cv::Mat() : frames[0]); + return (frames.empty() ? Matrix() : frames[0]); } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } - std::vector Producer::getFrames() + std::vector Producer::getFrames() { try { - std::vector frames; + std::vector frames; if (isOpened()) { @@ -144,7 +145,7 @@ namespace op } } - std::vector Producer::getCameraMatrices() + std::vector Producer::getCameraMatrices() { try { @@ -157,7 +158,7 @@ namespace op } } - std::vector Producer::getCameraExtrinsics() + std::vector Producer::getCameraExtrinsics() { try { @@ -170,7 +171,7 @@ namespace op } } - std::vector Producer::getCameraIntrinsics() + std::vector Producer::getCameraIntrinsics() { try { @@ -273,7 +274,7 @@ namespace op } } - void Producer::checkFrameIntegrity(cv::Mat& frame) + void Producer::checkFrameIntegrity(Matrix& frame) { try { @@ -290,15 +291,15 @@ namespace op mNumberEmptyFrames = 0; if (mType != ProducerType::ImageDirectory - && ((frame.cols != get(CV_CAP_PROP_FRAME_WIDTH) && get(CV_CAP_PROP_FRAME_WIDTH) > 0) - || (frame.rows != get(CV_CAP_PROP_FRAME_HEIGHT) && get(CV_CAP_PROP_FRAME_HEIGHT) > 0))) + && ((frame.cols() != get(CV_CAP_PROP_FRAME_WIDTH) && get(CV_CAP_PROP_FRAME_WIDTH) > 0) + || (frame.rows() != get(CV_CAP_PROP_FRAME_HEIGHT) && get(CV_CAP_PROP_FRAME_HEIGHT) > 0))) { log("Frame size changed. Returning empty frame.\nExpected vs. received sizes: " + std::to_string(positiveIntRound(get(CV_CAP_PROP_FRAME_WIDTH))) + "x" + std::to_string(positiveIntRound(get(CV_CAP_PROP_FRAME_HEIGHT))) - + " vs. " + std::to_string(frame.cols) + "x" + std::to_string(frame.rows), + + " vs. " + std::to_string(frame.cols()) + "x" + std::to_string(frame.rows()), Priority::Max, __LINE__, __FUNCTION__, __FILE__); - frame = cv::Mat(); + frame = Matrix(); } } } @@ -373,7 +374,7 @@ namespace op } else { - std::vector frames; + std::vector frames; for (auto i = 0 ; i < std::floor(difference) ; i++) frames = getRawFrames(); } diff --git a/src/openpose/producer/spinnakerWrapper.cpp b/src/openpose/producer/spinnakerWrapper.cpp index 8cd746c1..3771a776 100644 --- a/src/openpose/producer/spinnakerWrapper.cpp +++ b/src/openpose/producer/spinnakerWrapper.cpp @@ -1,7 +1,8 @@ -#include // OPEN_CV_IS_4_OR_HIGHER +#include #include #include #include // cv::undistort, cv::initUndistortRectifyMap +#include // OPEN_CV_IS_4_OR_HIGHER #ifdef OPEN_CV_IS_4_OR_HIGHER #include // cv::initUndistortRectifyMap for OpenCV 4 #endif @@ -9,7 +10,6 @@ #include #endif #include -#include namespace op { @@ -510,12 +510,15 @@ namespace op } // This function acquires and displays images from each device. - std::vector acquireImages(const std::vector& cameraIntrinsics, - const std::vector& cameraDistorsions, - const int cameraIndex = -1) + std::vector acquireImages( + const std::vector& opCameraIntrinsics, + const std::vector& opCameraDistorsions, + const int cameraIndex = -1) { try { + OP_OP2CVVECTORMAT(cameraIntrinsics, opCameraIntrinsics) + OP_OP2CVVECTORMAT(cameraDistorsions, opCameraDistorsions) // std::vector cvMats; // Retrieve, convert, and return an image for each camera @@ -639,7 +642,8 @@ namespace op mCvMats = std::vector{mCvMats[cameraIndex]}; } } - return mCvMats; + OP_CV2OPVECTORMAT(opMats, mCvMats) + return opMats; } catch (Spinnaker::Exception &e) { @@ -890,7 +894,7 @@ namespace op if (cvMats.empty()) error("Cameras could not be opened.", __LINE__, __FUNCTION__, __FILE__); // Get resolution - upImpl->mResolution = Point{cvMats[0].cols, cvMats[0].rows}; + upImpl->mResolution = Point{cvMats[0].cols(), cvMats[0].rows()}; const std::string numberCameras = std::to_string(upImpl->mCameraIndex < 0 ? serialNumbers.size() : 1); log("\nRunning for " + numberCameras + " out of " + std::to_string(serialNumbers.size()) @@ -925,7 +929,7 @@ namespace op } } - std::vector SpinnakerWrapper::getRawFrames() + std::vector SpinnakerWrapper::getRawFrames() { try { @@ -960,7 +964,7 @@ namespace op } } - std::vector SpinnakerWrapper::getCameraMatrices() const + std::vector SpinnakerWrapper::getCameraMatrices() const { try { @@ -977,7 +981,7 @@ namespace op } } - std::vector SpinnakerWrapper::getCameraExtrinsics() const + std::vector SpinnakerWrapper::getCameraExtrinsics() const { try { @@ -994,7 +998,7 @@ namespace op } } - std::vector SpinnakerWrapper::getCameraIntrinsics() const + std::vector SpinnakerWrapper::getCameraIntrinsics() const { try { diff --git a/src/openpose/producer/videoCaptureReader.cpp b/src/openpose/producer/videoCaptureReader.cpp index fae599bb..03df9397 100644 --- a/src/openpose/producer/videoCaptureReader.cpp +++ b/src/openpose/producer/videoCaptureReader.cpp @@ -1,14 +1,30 @@ +#include #include #include #include -#include +#include namespace op { + struct VideoCaptureReader::ImplVideoCaptureReader + { + cv::VideoCapture mVideoCapture; + + ImplVideoCaptureReader() + { + } + + ImplVideoCaptureReader(const std::string& path) : + mVideoCapture{path} + { + } + }; + VideoCaptureReader::VideoCaptureReader(const int index, const bool throwExceptionIfNoOpened, const std::string& cameraParameterPath, const bool undistortImage, const int numberViews) : - Producer{ProducerType::Webcam, cameraParameterPath, undistortImage, numberViews} + Producer{ProducerType::Webcam, cameraParameterPath, undistortImage, numberViews}, + upImpl{new ImplVideoCaptureReader{}} { try { @@ -24,7 +40,7 @@ namespace op const std::string& cameraParameterPath, const bool undistortImage, const int numberViews) : Producer{producerType, cameraParameterPath, undistortImage, numberViews}, - mVideoCapture{path} + upImpl{new ImplVideoCaptureReader{path}} { try { @@ -73,7 +89,7 @@ namespace op { try { - return mVideoCapture.isOpened(); + return upImpl->mVideoCapture.isOpened(); } catch (const std::exception& e) { @@ -82,43 +98,44 @@ namespace op } } - cv::Mat VideoCaptureReader::getRawFrame() + Matrix VideoCaptureReader::getRawFrame() { try { // Get frame cv::Mat frame; - mVideoCapture >> frame; + upImpl->mVideoCapture >> frame; // Skip frames if frame step > 1 const auto frameStep = Producer::get(ProducerProperty::FrameStep); if (frameStep > 1 && !frame.empty() && get(CV_CAP_PROP_POS_FRAMES) < get(CV_CAP_PROP_FRAME_COUNT)-1) { // Close if end of video if (get(CV_CAP_PROP_POS_FRAMES) + frameStep-1 >= get(CV_CAP_PROP_FRAME_COUNT)) - mVideoCapture.release(); + upImpl->mVideoCapture.release(); // Frame step usually more efficient if just reading sequentially else if (frameStep < 51) for (auto i = 1 ; i < frameStep ; i++) - mVideoCapture >> frame; + upImpl->mVideoCapture >> frame; // Using set(CV_CAP_PROP_POS_FRAMES, value) is efficient only if step is big else set(CV_CAP_PROP_POS_FRAMES, get(CV_CAP_PROP_POS_FRAMES) + frameStep-1); } // Return frame - return frame; + Matrix opFrame = OP_CV2OPMAT(frame); + return opFrame; } catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } - std::vector VideoCaptureReader::getRawFrames() + std::vector VideoCaptureReader::getRawFrames() { try { - return std::vector{getRawFrame()}; + return std::vector{getRawFrame()}; } catch (const std::exception& e) { @@ -132,7 +149,7 @@ namespace op try { // Open webcam - mVideoCapture = cv::VideoCapture{index}; + upImpl->mVideoCapture = cv::VideoCapture{index}; // Make sure video capture was opened if (throwExceptionIfNoOpened && !isOpened()) error("VideoCapture (webcam) could not be opened.", __LINE__, __FUNCTION__, __FILE__); @@ -147,9 +164,9 @@ namespace op { try { - if (mVideoCapture.isOpened()) + if (upImpl->mVideoCapture.isOpened()) { - mVideoCapture.release(); + upImpl->mVideoCapture.release(); log("cv::VideoCapture released.", Priority::Low, __LINE__, __FUNCTION__, __FILE__); } } @@ -170,13 +187,13 @@ namespace op && Producer::get(ProducerProperty::Rotation) != 180.)) { if (capProperty == CV_CAP_PROP_FRAME_WIDTH) - return mVideoCapture.get(CV_CAP_PROP_FRAME_HEIGHT); + return upImpl->mVideoCapture.get(CV_CAP_PROP_FRAME_HEIGHT); else - return mVideoCapture.get(CV_CAP_PROP_FRAME_WIDTH); + return upImpl->mVideoCapture.get(CV_CAP_PROP_FRAME_WIDTH); } // Generic cases - return mVideoCapture.get(capProperty); + return upImpl->mVideoCapture.get(capProperty); } catch (const std::exception& e) { @@ -189,7 +206,7 @@ namespace op { try { - mVideoCapture.set(capProperty, value); + upImpl->mVideoCapture.set(capProperty, value); } catch (const std::exception& e) { diff --git a/src/openpose/producer/videoReader.cpp b/src/openpose/producer/videoReader.cpp index 51051f4d..6f468556 100644 --- a/src/openpose/producer/videoReader.cpp +++ b/src/openpose/producer/videoReader.cpp @@ -1,6 +1,7 @@ +#include #include #include -#include +#include namespace op { @@ -57,7 +58,7 @@ namespace op } } - cv::Mat VideoReader::getRawFrame() + Matrix VideoReader::getRawFrame() { try { @@ -66,11 +67,11 @@ namespace op catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } - std::vector VideoReader::getRawFrames() + std::vector VideoReader::getRawFrames() { try { @@ -79,15 +80,20 @@ namespace op // Split image if (cvMats.size() == 1 && numberViews > 1) { - cv::Mat cvMatConcatenated = cvMats.at(0); + Matrix opMatConcatenated = cvMats.at(0); + cv::Mat matConcatenated = OP_OP2CVMAT(opMatConcatenated); cvMats.clear(); - const auto individualWidth = cvMatConcatenated.cols/numberViews; + const auto individualWidth = matConcatenated.cols/numberViews; for (auto i = 0 ; i < numberViews ; i++) - cvMats.emplace_back( - cv::Mat(cvMatConcatenated, - cv::Rect{(int)(i*individualWidth), 0, - (int)individualWidth, - (int)cvMatConcatenated.rows})); + { + cv::Mat cvMat( + matConcatenated, + cv::Rect{ + (int)(i*individualWidth), 0, + (int)individualWidth, (int)matConcatenated.rows }); + const Matrix opMat = OP_CV2OPMAT(cvMat); + cvMats.emplace_back(opMat); + } } // Sanity check else if (cvMats.size() != 1u && numberViews > 1) diff --git a/src/openpose/producer/webcamReader.cpp b/src/openpose/producer/webcamReader.cpp index 1e17de98..8fbc4b27 100644 --- a/src/openpose/producer/webcamReader.cpp +++ b/src/openpose/producer/webcamReader.cpp @@ -1,7 +1,8 @@ -#include +#include #include #include -#include +#include +#include namespace op { @@ -122,14 +123,14 @@ namespace op } } - cv::Mat WebcamReader::getRawFrame() + Matrix WebcamReader::getRawFrame() { try { mFrameNameCounter++; // Simple counter: 0,1,2,3,... // Retrieve frame from buffer - cv::Mat cvMat; + Matrix opMat; auto cvMatRetrieved = false; while (!cvMatRetrieved) { @@ -137,7 +138,7 @@ namespace op std::unique_lock lock{mBufferMutex}; if (!mBuffer.empty()) { - std::swap(cvMat, mBuffer); + std::swap(opMat, mBuffer); cvMatRetrieved = true; } // No frames available -> sleep & wait @@ -147,7 +148,7 @@ namespace op std::this_thread::sleep_for(std::chrono::microseconds{5}); } } - return cvMat; + return opMat; // Naive implementation - No flashing buffers // return VideoCaptureReader::getRawFrame(); @@ -155,15 +156,15 @@ namespace op catch (const std::exception& e) { error(e.what(), __LINE__, __FUNCTION__, __FILE__); - return cv::Mat(); + return Matrix(); } } - std::vector WebcamReader::getRawFrames() + std::vector WebcamReader::getRawFrames() { try { - return std::vector{getRawFrame()}; + return std::vector{getRawFrame()}; } catch (const std::exception& e) { @@ -185,14 +186,24 @@ namespace op if (mDisconnectedCounter > DISCONNETED_THRESHOLD) cameraConnected = reset(); // Get frame - auto cvMat = VideoCaptureReader::getRawFrame(); + auto opMat = VideoCaptureReader::getRawFrame(); // Detect whether camera is connected - const auto newNorm = ( - cvMat.empty() ? mLastNorm : cv::norm(cvMat.row(cvMat.rows/2))); + // Equivalent code: + // const auto newNorm = ( + // opMat.empty() ? mLastNorm : cv::norm(opMat.row(opMat.rows() / 2))); + double newNorm; + if (opMat.empty()) + newNorm = mLastNorm; + else + { + cv::Mat rowMat; + OP_CONST_MAT_RETURN_FUNCTION(rowMat, opMat, row(opMat.rows() / 2)); + newNorm = cv::norm(rowMat); + } if (mLastNorm == newNorm) { mDisconnectedCounter++; - if (mDisconnectedCounter > 1 && cvMat.empty()) + if (mDisconnectedCounter > 1 && opMat.empty()) log("Camera frame empty (it has occurred for the last " + std::to_string(mDisconnectedCounter) + " consecutive frames).", Priority::Max); } @@ -204,7 +215,7 @@ namespace op // If camera disconnected: black image if (!cameraConnected) { - cvMat = cv::Mat(mResolution.y, mResolution.x, CV_8UC3, cv::Scalar{0,0,0}); + cv::Mat cvMat(mResolution.y, mResolution.x, CV_8UC3, cv::Scalar{0,0,0}); putTextOnCvMat(cvMat, "Camera disconnected, reconnecting...", {cvMat.cols/16, cvMat.rows/2}, cv::Scalar{255, 255, 255}, false, positiveIntRound(2.3*cvMat.cols)); // Anti flip + anti rotate frame (so it is balanced with the final flip + rotate) @@ -213,13 +224,14 @@ namespace op if (int(std::round(rotationAngle)) % 180 != 0.) rotationAngle = 0; const auto flipFrame = ((unsigned char)Producer::get(ProducerProperty::Flip) == 1.); - rotateAndFlipFrame(cvMat, rotationAngle, flipFrame); + opMat = OP_CV2OPMAT(cvMat); + rotateAndFlipFrame(opMat, rotationAngle, flipFrame); } // Move to buffer - if (!cvMat.empty()) + if (!opMat.empty()) { const std::lock_guard lock{mBufferMutex}; - std::swap(mBuffer, cvMat); + std::swap(mBuffer, opMat); } } } diff --git a/src/openpose/thread/CMakeLists.txt b/src/openpose/thread/CMakeLists.txt index 50151397..41009ced 100644 --- a/src/openpose/thread/CMakeLists.txt +++ b/src/openpose/thread/CMakeLists.txt @@ -7,13 +7,13 @@ set(SOURCES_OP_THREAD_WITH_CP ${SOURCES_OP_THREAD_WITH_CP} PARENT_SCOPE) set(SOURCES_OPENPOSE ${SOURCES_OPENPOSE} ${SOURCES_OP_THREAD_WITH_CP} PARENT_SCOPE) if (UNIX OR APPLE) - add_library(openpose_thread ${SOURCES_OP_THREAD}) + add_library(openpose_thread ${SOURCES_OP_THREAD}) target_link_libraries(openpose_thread openpose_core) - - install(TARGETS openpose_thread - EXPORT OpenPose - RUNTIME DESTINATION bin - LIBRARY DESTINATION lib - ARCHIVE DESTINATION lib/openpose) + + install(TARGETS openpose_thread + EXPORT OpenPose + RUNTIME DESTINATION bin + LIBRARY DESTINATION lib + ARCHIVE DESTINATION lib/openpose) endif (UNIX OR APPLE) diff --git a/src/openpose/tracking/CMakeLists.txt b/src/openpose/tracking/CMakeLists.txt index 7cff327b..fab434b8 100644 --- a/src/openpose/tracking/CMakeLists.txt +++ b/src/openpose/tracking/CMakeLists.txt @@ -11,13 +11,13 @@ set(SOURCES_OP_TRACKING_WITH_CP ${SOURCES_OP_TRACKING_WITH_CP} PARENT_SCOPE) set(SOURCES_OPENPOSE ${SOURCES_OPENPOSE} ${SOURCES_OP_TRACKING_WITH_CP} PARENT_SCOPE) if (UNIX OR APPLE) - add_library(openpose_tracking ${SOURCES_OP_TRACKING}) + add_library(openpose_tracking ${SOURCES_OP_TRACKING}) target_link_libraries(openpose_tracking openpose_core) - install(TARGETS openpose_tracking - EXPORT OpenPose - RUNTIME DESTINATION bin - LIBRARY DESTINATION lib - ARCHIVE DESTINATION lib/openpose) + install(TARGETS openpose_tracking + EXPORT OpenPose + RUNTIME DESTINATION bin + LIBRARY DESTINATION lib + ARCHIVE DESTINATION lib/openpose) endif (UNIX OR APPLE) diff --git a/src/openpose/tracking/personIdExtractor.cpp b/src/openpose/tracking/personIdExtractor.cpp index 9bc4cabf..db44591d 100644 --- a/src/openpose/tracking/personIdExtractor.cpp +++ b/src/openpose/tracking/personIdExtractor.cpp @@ -1,11 +1,30 @@ -#include -#include #include +#include +#include +#include +#include +#include +#include // #define LK_CUDA namespace op { + struct PersonEntry + { + long long counterLastDetection; + std::vector keypoints; + std::vector status; + /* + PersonEntry(long long _last_frame, + std::vector _keypoints, + std::vector _active): + last_frame(_last_frame), keypoints(_keypoints), + active(_active) + {} + */ + }; + const std::string errorMessage = "ID extractor function (`--identification` flag) not implemented" " for multiple-view processing."; @@ -46,9 +65,9 @@ namespace op keypoints.emplace_back(cp); if (poseKeypoints[{p,kp,2}] < confidenceThreshold) - status.emplace_back(1); + status.emplace_back(char(1)); else - status.emplace_back(0); + status.emplace_back(char(0)); } } // Return result @@ -129,9 +148,9 @@ namespace op keypoints.emplace_back(cp); if (poseKeypoints[{p,kp,2}] < confidenceThreshold) - status.emplace_back(1); + status.emplace_back(char(1)); else - status.emplace_back(0); + status.emplace_back(char(0)); } } } @@ -361,14 +380,36 @@ namespace op // } // } + struct PersonIdExtractor::ImplPersonIdExtractor + { + const float mConfidenceThreshold; + const float mInlierRatioThreshold; + const float mDistanceThreshold; + const int mNumberFramesToDeletePerson; + long long mNextPersonId; + cv::Mat mImagePrevious; + std::vector mPyramidImagesPrevious; + std::unordered_map mPersonEntries; + // Thread-safe variables + std::atomic mLastFrameId; + + ImplPersonIdExtractor( + const float confidenceThreshold, const float inlierRatioThreshold, const float distanceThreshold, + const int numberFramesToDeletePerson) : + mConfidenceThreshold{confidenceThreshold}, + mInlierRatioThreshold{inlierRatioThreshold}, + mDistanceThreshold{distanceThreshold}, + mNumberFramesToDeletePerson{numberFramesToDeletePerson}, + mNextPersonId{0ll}, + mLastFrameId{-1ll} + { + } + }; + PersonIdExtractor::PersonIdExtractor(const float confidenceThreshold, const float inlierRatioThreshold, const float distanceThreshold, const int numberFramesToDeletePerson) : - mConfidenceThreshold{confidenceThreshold}, - mInlierRatioThreshold{inlierRatioThreshold}, - mDistanceThreshold{distanceThreshold}, - mNumberFramesToDeletePerson{numberFramesToDeletePerson}, - mNextPersonId{0ll}, - mLastFrameId{-1ll} + spImpl{new ImplPersonIdExtractor{confidenceThreshold, inlierRatioThreshold, distanceThreshold, + numberFramesToDeletePerson}} { try { @@ -385,7 +426,7 @@ namespace op { } - Array PersonIdExtractor::extractIds(const Array& poseKeypoints, const cv::Mat& cvMatInput, + Array PersonIdExtractor::extractIds(const Array& poseKeypoints, const Matrix& cvMatInput, const unsigned long long imageViewIndex) { try @@ -396,34 +437,34 @@ namespace op // Result initialization Array poseIds; - const auto openposePersonEntries = captureKeypoints(poseKeypoints, mConfidenceThreshold); -// log(mPersonEntries.size()); + const auto openposePersonEntries = captureKeypoints(poseKeypoints, spImpl->mConfidenceThreshold); // First frame - if (mImagePrevious.empty()) + const cv::Mat cvMatcvMatInput = OP_OP2CVCONSTMAT(cvMatInput); + if (spImpl->mImagePrevious.empty()) { // Add first persons to the LK set - initializeLK(mPersonEntries, mNextPersonId, poseKeypoints, mConfidenceThreshold); + initializeLK(spImpl->mPersonEntries, spImpl->mNextPersonId, poseKeypoints, spImpl->mConfidenceThreshold); // Capture current frame as floating point - cvMatInput.convertTo(mImagePrevious, CV_32F); + cvMatcvMatInput.convertTo(spImpl->mImagePrevious, CV_32F); } // Rest else { cv::Mat imageCurrent; std::vector pyramidImagesCurrent; - cvMatInput.convertTo(imageCurrent, CV_32F); - updateLK(mPersonEntries, mPyramidImagesPrevious, pyramidImagesCurrent, mImagePrevious, imageCurrent, - mNumberFramesToDeletePerson); - mImagePrevious = imageCurrent; - mPyramidImagesPrevious = pyramidImagesCurrent; + cvMatcvMatInput.convertTo(imageCurrent, CV_32F); + updateLK(spImpl->mPersonEntries, spImpl->mPyramidImagesPrevious, pyramidImagesCurrent, spImpl->mImagePrevious, imageCurrent, + spImpl->mNumberFramesToDeletePerson); + spImpl->mImagePrevious = imageCurrent; + spImpl->mPyramidImagesPrevious = pyramidImagesCurrent; } // Get poseIds and update LKset according to OpenPose set // poseIds = matchLKAndOP( poseIds = matchLKAndOPGreedy( - mPersonEntries, mNextPersonId, openposePersonEntries, mImagePrevious, mInlierRatioThreshold, - mDistanceThreshold); + spImpl->mPersonEntries, spImpl->mNextPersonId, openposePersonEntries, spImpl->mImagePrevious, spImpl->mInlierRatioThreshold, + spImpl->mDistanceThreshold); return poseIds; } @@ -435,7 +476,7 @@ namespace op } Array PersonIdExtractor::extractIdsLockThread(const Array& poseKeypoints, - const cv::Mat& cvMatInput, + const Matrix& cvMatInput, const unsigned long long imageViewIndex, const long long frameId) { @@ -445,12 +486,12 @@ namespace op if (imageViewIndex > 0) error(errorMessage, __LINE__, __FUNCTION__, __FILE__); // Wait for desired order - while (mLastFrameId < frameId - 1) + while (spImpl->mLastFrameId < frameId - 1) std::this_thread::sleep_for(std::chrono::microseconds{100}); // Extract IDs const auto ids = extractIds(poseKeypoints, cvMatInput, imageViewIndex); // Update last frame id - mLastFrameId = frameId; + spImpl->mLastFrameId = frameId; // Return person ids return ids; } diff --git a/src/openpose/tracking/personTracker.cpp b/src/openpose/tracking/personTracker.cpp index ab3506fd..a5880480 100644 --- a/src/openpose/tracking/personTracker.cpp +++ b/src/openpose/tracking/personTracker.cpp @@ -1,11 +1,32 @@ -#include -#include // cv::resize #include +#include +#include +#include #include -#include +#include +#include namespace op { + struct PersonTrackerEntry + { + std::vector keypoints; + std::vector lastKeypoints; + std::vector status; + std::vector getPredicted() const + { + std::vector predictedKeypoints(keypoints); + if (!lastKeypoints.size()) + return predictedKeypoints; + for (size_t i=0; i mPyramidImagesPrevious; + std::unordered_map mPersonEntries; + Array mLastPoseIds; + + // Thread-safe variables + std::atomic mLastFrameId; + + ImplPersonTracker( + const bool mergeResults, const int levels, const int patchSize, const float confidenceThreshold, + const bool trackVelocity, const bool scaleVarying, const float rescale) : + mMergeResults{mergeResults}, + mLevels{levels}, + mPatchSize{patchSize}, + mTrackVelocity{trackVelocity}, + mConfidenceThreshold{confidenceThreshold}, + mScaleVarying{scaleVarying}, + mRescale{rescale}, + mLastFrameId{-1ll} + { + } + }; + + PersonTracker::PersonTracker( + const bool mergeResults, const int levels, const int patchSize, const float confidenceThreshold, + const bool trackVelocity, const bool scaleVarying, const float rescale) : + spImpl{new ImplPersonTracker{mergeResults, levels, patchSize, confidenceThreshold, trackVelocity, + scaleVarying, rescale}} { try { @@ -371,7 +419,7 @@ namespace op } void PersonTracker::track(Array& poseKeypoints, Array& poseIds, - const cv::Mat& cvMatInput) + const Matrix& cvMatInput) { try { @@ -402,7 +450,7 @@ namespace op // if mMergeResults == true --> Combine OP + LK tracker // if mMergeResults == false --> Run LK tracker ONLY IF poseKeypoints.empty() - bool mergeResults = mMergeResults; + bool mergeResults = spImpl->mMergeResults; mergeResults = true; // Sanity Checks @@ -411,22 +459,23 @@ namespace op __LINE__, __FUNCTION__, __FILE__); // First frame - if (mImagePrevious.empty()) + const cv::Mat cvMatcvMatInput = OP_OP2CVCONSTMAT(cvMatInput); + if (spImpl->mImagePrevious.empty()) { // Create mPersonEntries - personEntriesFromOP(mPersonEntries, poseKeypoints, poseIds, mConfidenceThreshold); + personEntriesFromOP(spImpl->mPersonEntries, poseKeypoints, poseIds, spImpl->mConfidenceThreshold); // Capture current frame as floating point - cvMatInput.convertTo(mImagePrevious, CV_8UC3); + cvMatcvMatInput.convertTo(spImpl->mImagePrevious, CV_8UC3); // Rescale - if (mRescale) + if (spImpl->mRescale) { cv::Size rescaleSize{ - positiveIntRound(mRescale), - positiveIntRound(mImagePrevious.size().height/(mImagePrevious.size().width/mRescale))}; - cv::resize(mImagePrevious, mImagePrevious, rescaleSize, 0, 0, cv::INTER_CUBIC); + positiveIntRound(spImpl->mRescale), + positiveIntRound(spImpl->mImagePrevious.size().height/(spImpl->mImagePrevious.size().width/ spImpl->mRescale))}; + cv::resize(spImpl->mImagePrevious, spImpl->mImagePrevious, rescaleSize, 0, 0, cv::INTER_CUBIC); } // Save Last Ids - mLastPoseIds = poseIds.clone(); + spImpl->mLastPoseIds = poseIds.clone(); } // Any other frame else @@ -437,43 +486,43 @@ namespace op { cv::Mat imageCurrent; std::vector pyramidImagesCurrent; - cvMatInput.convertTo(imageCurrent, CV_8UC3); + cvMatcvMatInput.convertTo(imageCurrent, CV_8UC3); float xScale = 1., yScale = 1.; - if (mRescale) + if (spImpl->mRescale) { cv::Size rescaleSize{ - positiveIntRound(mRescale), - positiveIntRound(imageCurrent.size().height/(imageCurrent.size().width/mRescale))}; + positiveIntRound(spImpl->mRescale), + positiveIntRound(imageCurrent.size().height/(imageCurrent.size().width/ spImpl->mRescale))}; xScale = imageCurrent.size().width / (float)rescaleSize.width; yScale = imageCurrent.size().height / (float)rescaleSize.height; cv::resize(imageCurrent, imageCurrent, rescaleSize, 0, 0, cv::INTER_CUBIC); } - scaleKeypoints(mPersonEntries, 1.f/xScale, 1.f/yScale); - updateLK(mPersonEntries, mPyramidImagesPrevious, pyramidImagesCurrent, mImagePrevious, - imageCurrent, mLevels, mPatchSize, mTrackVelocity, mScaleVarying); - scaleKeypoints(mPersonEntries, xScale, yScale); - mImagePrevious = imageCurrent; - mPyramidImagesPrevious = pyramidImagesCurrent; + scaleKeypoints(spImpl->mPersonEntries, 1.f/xScale, 1.f/yScale); + updateLK(spImpl->mPersonEntries, spImpl->mPyramidImagesPrevious, pyramidImagesCurrent, spImpl->mImagePrevious, + imageCurrent, spImpl->mLevels, spImpl->mPatchSize, spImpl->mTrackVelocity, spImpl->mScaleVarying); + scaleKeypoints(spImpl->mPersonEntries, xScale, yScale); + spImpl->mImagePrevious = imageCurrent; + spImpl->mPyramidImagesPrevious = pyramidImagesCurrent; } // There is new OP Data if (newOPData) { - mLastPoseIds = poseIds.clone(); - syncPersonEntriesWithOP(mPersonEntries, poseKeypoints, mLastPoseIds, mConfidenceThreshold, + spImpl->mLastPoseIds = poseIds.clone(); + syncPersonEntriesWithOP(spImpl->mPersonEntries, poseKeypoints, spImpl->mLastPoseIds, spImpl->mConfidenceThreshold, mergeResults); - opFromPersonEntries(poseKeypoints, mPersonEntries, mLastPoseIds); + opFromPersonEntries(poseKeypoints, spImpl->mPersonEntries, spImpl->mLastPoseIds); } // There is no new OP Data else { - opFromPersonEntries(poseKeypoints, mPersonEntries, mLastPoseIds); - poseIds = mLastPoseIds.clone(); + opFromPersonEntries(poseKeypoints, spImpl->mPersonEntries, spImpl->mLastPoseIds); + poseIds = spImpl->mLastPoseIds.clone(); } } // cv::Mat debugImage = cvMatInput.clone(); - // vizPersonEntries(debugImage, mPersonEntries, mTrackVelocity); + // vizPersonEntries(debugImage, spImpl->mPersonEntries, spImpl->mTrackVelocity); // cv::imshow("win", debugImage); // cv::waitKey(15); } @@ -484,17 +533,17 @@ namespace op } void PersonTracker::trackLockThread(Array& poseKeypoints, Array& poseIds, - const cv::Mat& cvMatInput, const long long frameId) + const Matrix& cvMatInput, const long long frameId) { try { // Wait for desired order - while (mLastFrameId < frameId - 1) + while (spImpl->mLastFrameId < frameId - 1) std::this_thread::sleep_for(std::chrono::microseconds{100}); // Extract IDs track(poseKeypoints, poseIds, cvMatInput); // Update last frame id - mLastFrameId = frameId; + spImpl->mLastFrameId = frameId; } catch (const std::exception& e) { @@ -506,7 +555,7 @@ namespace op { try { - return mMergeResults; + return spImpl->mMergeResults; } catch (const std::exception& e) { diff --git a/src/openpose/tracking/pyramidalLK.cpp b/src/openpose/tracking/pyramidalLK.cpp index d6f3100a..41affc73 100644 --- a/src/openpose/tracking/pyramidalLK.cpp +++ b/src/openpose/tracking/pyramidalLK.cpp @@ -1,3 +1,4 @@ +#include #ifdef WITH_SSE4 #include #include "smmintrin.h" @@ -12,7 +13,6 @@ #include // cv::pyrDown #include // cv::buildOpticalFlowPyramid #include -#include //#define DEBUG // #ifdef DEBUG @@ -67,37 +67,36 @@ namespace op } #endif -#ifdef WITH_AVX - +// Function aligned_alloc requires C++17 in VS +#if defined (WITH_AVX) && !defined (_WIN32) float avx_dot_product(std::vector &av, std::vector &bv) { + /* Get SIMD-vector pointers to the start of each vector */ + const size_t niters = av.size() / 8; - /* Get SIMD-vector pointers to the start of each vector */ - unsigned int niters = av.size() / 8; + float *a = (float *)aligned_alloc(32, av.size() * sizeof(float)); + float *b = (float *)aligned_alloc(32, av.size() * sizeof(float)); + memcpy(a, &av[0], av.size() * sizeof(float)); + memcpy(b, &bv[0], bv.size() * sizeof(float)); - float *a = (float *) aligned_alloc(32, av.size()*sizeof(float)); - float *b = (float *) aligned_alloc(32, av.size()*sizeof(float)); - memcpy(a,&av[0],av.size()*sizeof(float)); - memcpy(b,&bv[0],bv.size()*sizeof(float)); + __m256 *ptrA = (__m256*) &a[0], *ptrB = (__m256*) &b[0]; + __m256 res = _mm256_set1_ps(0.0); - __m256 *ptrA = (__m256*) &a[0], *ptrB = (__m256*) &b[0]; - __m256 res = _mm256_set1_ps(0.0); + for (size_t i = 0; i < niters; i++, ptrA++, ptrB++) + res = _mm256_add_ps(_mm256_dp_ps(*ptrA, *ptrB, 255), res); - for (unsigned int i = 0; i < niters; i++, ptrA++,ptrB++) - res = _mm256_add_ps(_mm256_dp_ps(*ptrA, *ptrB, 255), res); + /* Get result back from the SIMD vector */ + float fres[8]; + _mm256_storeu_ps(fres, res); + const size_t q = 8 * niters; - /* Get result back from the SIMD vector */ - float fres[8]; - _mm256_storeu_ps (fres, res); - int q = 8 * niters; + for (size_t i = 0; i < av.size() % 8; i++) + fres[0] += (a[i + q] * b[i + q]); - for (unsigned int i = 0; i < av.size() % 8; i++) - fres[0] += (a[i+q]*b[i+q]); + free(a); + free(b); - free(a); - free(b); - - return fres[0] + fres[4]; + return fres[0] + fres[4]; } #endif @@ -107,32 +106,31 @@ namespace op try { // Calculate sums +#if defined (WITH_AVX) && !defined (_WIN32) + const float sumXX = avx_dot_product(ix,ix); + const float sumYY = avx_dot_product(iy,iy); + const float sumXY = avx_dot_product(ix,iy); + const float sumXT = avx_dot_product(ix,it); + const float sumYT = avx_dot_product(iy,it); +#elif defined (WITH_SSE4) + const float sumXX = sse_dot_product(ix,ix); + const float sumYY = sse_dot_product(iy,iy); + const float sumXY = sse_dot_product(ix,iy); + const float sumXT = sse_dot_product(ix,it); + const float sumYT = sse_dot_product(iy,it); +#else auto sumXX = 0.f; auto sumYY = 0.f; auto sumXT = 0.f; auto sumYT = 0.f; auto sumXY = 0.f; - -#ifdef WITH_AVX - sumXX = avx_dot_product(ix,ix); - sumYY = avx_dot_product(iy,iy); - sumXY = avx_dot_product(ix,iy); - sumXT = avx_dot_product(ix,it); - sumYT = avx_dot_product(iy,it); -#elif defined (WITH_SSE4) - sumXX = sse_dot_product(ix,ix); - sumYY = sse_dot_product(iy,iy); - sumXY = sse_dot_product(ix,iy); - sumXT = sse_dot_product(ix,it); - sumYT = sse_dot_product(iy,it); -#else for (auto i = 0u; i < ix.size(); i++) { - sumXX += ix[i] * ix[i]; - sumYY += iy[i] * iy[i]; - sumXY += ix[i] * iy[i]; - sumXT += ix[i] * it[i]; - sumYT += iy[i] * it[i]; + sumXX += ix[i] * ix[i]; + sumYY += iy[i] * iy[i]; + sumXY += ix[i] * iy[i]; + sumXT += ix[i] * it[i]; + sumYT += iy[i] * it[i]; } #endif diff --git a/src/openpose/tracking/pyramidalLK.cu b/src/openpose/tracking/pyramidalLK.cu index e88c7c41..29c3617d 100644 --- a/src/openpose/tracking/pyramidalLK.cu +++ b/src/openpose/tracking/pyramidalLK.cu @@ -1,3 +1,4 @@ +#include #ifdef WITH_TRACKING #include #include @@ -16,7 +17,6 @@ #define cvCuda cv::cuda #endif #endif -#include // Error codes for kernel caller #define IMAGE_SIZES_NEQUAL -1 diff --git a/src/openpose/unity/unityBinding.cpp b/src/openpose/unity/unityBinding.cpp index 24d26ed1..0896d1d1 100644 --- a/src/openpose/unity/unityBinding.cpp +++ b/src/openpose/unity/unityBinding.cpp @@ -670,22 +670,22 @@ namespace op } } - OP_API void _OPConfigureDebugging( - uchar loggingLevel, // Priority - bool disableMultiThread, - unsigned long long profileSpeed) - { - try - { - ConfigureLog::setPriorityThreshold((Priority)loggingLevel); - sMultiThreadDisabled = disableMultiThread; - Profiler::setDefaultX(profileSpeed); - } - catch (const std::exception& e) - { - errorDestructor(e.what(), __LINE__, __FUNCTION__, __FILE__); - } - } + OP_API void _OPConfigureDebugging( + uchar loggingLevel, // Priority + bool disableMultiThread, + unsigned long long profileSpeed) + { + try + { + ConfigureLog::setPriorityThreshold((Priority)loggingLevel); + sMultiThreadDisabled = disableMultiThread; + Profiler::setDefaultX(profileSpeed); + } + catch (const std::exception& e) + { + errorDestructor(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } } } #endif diff --git a/src/openpose/utilities/CMakeLists.txt b/src/openpose/utilities/CMakeLists.txt index 46ccede0..2c8ece1e 100644 --- a/src/openpose/utilities/CMakeLists.txt +++ b/src/openpose/utilities/CMakeLists.txt @@ -4,6 +4,7 @@ set(SOURCES_OP_UTILITIES flagsToOpenPose.cpp keypoint.cpp openCv.cpp + openCvPrivate.cpp profiler.cpp string.cpp) diff --git a/src/openpose/utilities/fileSystem.cpp b/src/openpose/utilities/fileSystem.cpp index 0c3d941c..f1ced9a8 100644 --- a/src/openpose/utilities/fileSystem.cpp +++ b/src/openpose/utilities/fileSystem.cpp @@ -1,4 +1,8 @@ -#include // fopen +#include +#include // std::replace +#include // std::isdigit +#include // std::fopen +#include // std::strncmp #ifdef _WIN32 #include // _mkdir #include // DWORD, GetFileAttributesA @@ -9,7 +13,6 @@ #error Unknown environment! #endif #include -#include namespace op { diff --git a/src/openpose/utilities/keypoint.cpp b/src/openpose/utilities/keypoint.cpp index 3e618276..ec6650a9 100644 --- a/src/openpose/utilities/keypoint.cpp +++ b/src/openpose/utilities/keypoint.cpp @@ -185,16 +185,17 @@ namespace op { // Array --> cv::Mat auto frame = frameArray.getCvMat(); + cv::Mat cvFrame = OP_OP2CVMAT(frame); // Sanity check - if (frame.channels() != 3) + if (cvFrame.channels() != 3) error(errorMessage, __LINE__, __FUNCTION__, __FILE__); // Get frame channels - const auto width = frame.size[1]; - const auto height = frame.size[0]; + const auto width = cvFrame.size[1]; + const auto height = cvFrame.size[0]; const auto area = width * height; - cv::Mat frameBGR(height, width, CV_32FC3, frame.data); + cv::Mat frameBGR(height, width, CV_32FC3, cvFrame.data); // Parameters const auto lineType = 8; diff --git a/src/openpose/utilities/openCv.cpp b/src/openpose/utilities/openCv.cpp index d56e97f4..b6fe86c6 100644 --- a/src/openpose/utilities/openCv.cpp +++ b/src/openpose/utilities/openCv.cpp @@ -1,36 +1,11 @@ -#include -#include #include +#include +#include +#include namespace op { - void putTextOnCvMat(cv::Mat& cvMat, const std::string& textToDisplay, const Point& position, - const cv::Scalar& color, const bool normalizeWidth, const int imageWidth) - { - try - { - const auto font = cv::FONT_HERSHEY_SIMPLEX; - const auto ratio = imageWidth/1280.; - // const auto fontScale = 0.75; - const auto fontScale = 0.8 * ratio; - const auto fontThickness = std::max(1, positiveIntRound(2*ratio)); - const auto shadowOffset = std::max(1, positiveIntRound(2*ratio)); - int baseline = 0; - const auto textSize = cv::getTextSize(textToDisplay, font, fontScale, fontThickness, &baseline); - const cv::Size finalPosition{position.x - (normalizeWidth ? textSize.width : 0), - position.y + textSize.height/2}; - cv::putText(cvMat, textToDisplay, - cv::Size{finalPosition.width + shadowOffset, finalPosition.height + shadowOffset}, - font, fontScale, cv::Scalar{0,0,0}, fontThickness); - cv::putText(cvMat, textToDisplay, finalPosition, font, fontScale, color, fontThickness); - } - catch (const std::exception& e) - { - error(e.what(), __LINE__, __FUNCTION__, __FILE__); - } - } - - void unrollArrayToUCharCvMat(cv::Mat& cvMatResult, const Array& array) + void unrollArrayToUCharCvMat(Matrix& matResult, const Array& array) { try { @@ -46,6 +21,7 @@ namespace op const auto areaInput = height * width; const auto areaOutput = channels * width; // Allocate cv::Mat if it was not initialized yet + cv::Mat cvMatResult = OP_OP2CVMAT(matResult); if (cvMatResult.empty() || cvMatResult.cols != channels * width || cvMatResult.rows != height) cvMatResult = cv::Mat(height, areaOutput, CV_8UC1); // Fill cvMatResult from array @@ -67,9 +43,10 @@ namespace op } } } + matResult = OP_CV2OPMAT(cvMatResult); } else - cvMatResult = cv::Mat(); + matResult = Matrix(); } catch (const std::exception& e) { @@ -77,10 +54,11 @@ namespace op } } - void uCharCvMatToFloatPtr(float* floatPtrImage, const cv::Mat& cvImage, const int normalize) + void uCharCvMatToFloatPtr(float* floatPtrImage, const Matrix& matImage, const int normalize) { try { + const cv::Mat cvImage = OP_OP2CVCONSTMAT(matImage); // float* (deep net format): C x H x W // cv::Mat (OpenCV format): H x W x C const int width = cvImage.cols; @@ -216,28 +194,7 @@ namespace op } } - void resizeFixedAspectRatio(cv::Mat& resizedCvMat, const cv::Mat& cvMat, const double scaleFactor, - const Point& targetSize, const int borderMode, const cv::Scalar& borderValue) - { - try - { - const cv::Size cvTargetSize{targetSize.x, targetSize.y}; - cv::Mat M = cv::Mat::eye(2,3,CV_64F); - M.at(0,0) = scaleFactor; - M.at(1,1) = scaleFactor; - if (scaleFactor != 1. || cvTargetSize != cvMat.size()) - cv::warpAffine(cvMat, resizedCvMat, M, cvTargetSize, - (scaleFactor > 1. ? cv::INTER_CUBIC : cv::INTER_AREA), borderMode, borderValue); - else - cvMat.copyTo(resizedCvMat); - } - catch (const std::exception& e) - { - error(e.what(), __LINE__, __FUNCTION__, __FILE__); - } - } - - void keepRoiInside(cv::Rect& roi, const int imageWidth, const int imageHeight) + void keepRoiInside(Rectangle& roi, const int imageWidth, const int imageHeight) { try { @@ -267,38 +224,39 @@ namespace op } } - void rotateAndFlipFrame(cv::Mat& frame, const double rotationAngle, const bool flipFrame) + void rotateAndFlipFrame(Matrix& frame, const double rotationAngle, const bool flipFrame) { try { - if (!frame.empty()) + cv::Mat cvMatFrame = OP_OP2CVMAT(frame); + if (!cvMatFrame.empty()) { const auto rotationAngleInt = (int)std::round(rotationAngle) % 360; if (rotationAngleInt == 0 || rotationAngleInt == 360) { if (flipFrame) - cv::flip(frame, frame, 1); + cv::flip(cvMatFrame, cvMatFrame, 1); } else if (rotationAngleInt == 90 || rotationAngleInt == -270) { - cv::transpose(frame, frame); + cv::transpose(cvMatFrame, cvMatFrame); if (!flipFrame) - cv::flip(frame, frame, 0); + cv::flip(cvMatFrame, cvMatFrame, 0); } else if (rotationAngleInt == 180 || rotationAngleInt == -180) { if (flipFrame) - cv::flip(frame, frame, 0); + cv::flip(cvMatFrame, cvMatFrame, 0); else - cv::flip(frame, frame, -1); + cv::flip(cvMatFrame, cvMatFrame, -1); } else if (rotationAngleInt == 270 || rotationAngleInt == -90) { - cv::transpose(frame, frame); + cv::transpose(cvMatFrame, cvMatFrame); if (flipFrame) - cv::flip(frame, frame, -1); + cv::flip(cvMatFrame, cvMatFrame, -1); else - cv::flip(frame, frame, 1); + cv::flip(cvMatFrame, cvMatFrame, 1); } else error("Rotation angle = " + std::to_string(rotationAngleInt) @@ -310,4 +268,108 @@ namespace op error(e.what(), __LINE__, __FUNCTION__, __FILE__); } } + + int getCvCapPropFrameCount() + { + try + { + return CV_CAP_PROP_FRAME_COUNT; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int getCvCapPropFrameFps() + { + try + { + return CV_CAP_PROP_FPS; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int getCvCapPropFrameWidth() + { + try + { + return CV_CAP_PROP_FRAME_WIDTH; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int getCvCapPropFrameHeight() + { + try + { + return CV_CAP_PROP_FRAME_HEIGHT; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int getCvFourcc(const char c1, const char c2, const char c3, const char c4) + { + try + { + return CV_FOURCC(c1,c2,c3,c4); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int getCvImwriteJpegQuality() + { + try + { + return CV_IMWRITE_JPEG_QUALITY; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int getCvImwritePngCompression() + { + try + { + return CV_IMWRITE_PNG_COMPRESSION; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } + + int getCvLoadImageAnydepth() + { + try + { + return CV_LOAD_IMAGE_ANYDEPTH; + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return -1; + } + } } diff --git a/src/openpose/utilities/openCvPrivate.cpp b/src/openpose/utilities/openCvPrivate.cpp new file mode 100644 index 00000000..0188a6cd --- /dev/null +++ b/src/openpose/utilities/openCvPrivate.cpp @@ -0,0 +1,54 @@ +#include +#include +#include +#include + +namespace op +{ + void putTextOnCvMat(cv::Mat& cvMat, const std::string& textToDisplay, const Point& position, + const cv::Scalar& color, const bool normalizeWidth, const int imageWidth) + { + try + { + const auto font = cv::FONT_HERSHEY_SIMPLEX; + const auto ratio = imageWidth/1280.; + // const auto fontScale = 0.75; + const auto fontScale = 0.8 * ratio; + const auto fontThickness = std::max(1, positiveIntRound(2*ratio)); + const auto shadowOffset = std::max(1, positiveIntRound(2*ratio)); + int baseline = 0; + const auto textSize = cv::getTextSize(textToDisplay, font, fontScale, fontThickness, &baseline); + const cv::Size finalPosition{position.x - (normalizeWidth ? textSize.width : 0), + position.y + textSize.height/2}; + cv::putText(cvMat, textToDisplay, + cv::Size{finalPosition.width + shadowOffset, finalPosition.height + shadowOffset}, + font, fontScale, cv::Scalar{0,0,0}, fontThickness); + cv::putText(cvMat, textToDisplay, finalPosition, font, fontScale, color, fontThickness); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } + + void resizeFixedAspectRatio(cv::Mat& resizedCvMat, const cv::Mat& cvMat, const double scaleFactor, + const Point& targetSize, const int borderMode, const cv::Scalar& borderValue) + { + try + { + const cv::Size cvTargetSize{targetSize.x, targetSize.y}; + cv::Mat M = cv::Mat::eye(2,3,CV_64F); + M.at(0,0) = scaleFactor; + M.at(1,1) = scaleFactor; + if (scaleFactor != 1. || cvTargetSize != cvMat.size()) + cv::warpAffine(cvMat, resizedCvMat, M, cvTargetSize, + (scaleFactor > 1. ? cv::INTER_CUBIC : cv::INTER_AREA), borderMode, borderValue); + else + cvMat.copyTo(resizedCvMat); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + } + } +} diff --git a/src/openpose/utilities/string.cpp b/src/openpose/utilities/string.cpp index 9b432128..c6263616 100644 --- a/src/openpose/utilities/string.cpp +++ b/src/openpose/utilities/string.cpp @@ -1,5 +1,7 @@ -#include // std::transform #include +#include // std::transform +#include // std::tolower, std::toupper +#include // std::tolower, std::toupper namespace op { @@ -88,8 +90,9 @@ namespace op { try { - auto result = string; - std::transform(string.begin(), string.end(), result.begin(), tolower); + std::string result = string; + std::transform(string.begin(), string.end(), result.begin(), + [](unsigned char c) { return (unsigned char)std::tolower(c); }); return result; } catch (const std::exception& e) @@ -103,8 +106,9 @@ namespace op { try { - auto result = string; - std::transform(string.begin(), string.end(), result.begin(), toupper); + std::string result = string; + std::transform(string.begin(), string.end(), result.begin(), + [](unsigned char c) { return (unsigned char)std::toupper(c); }); return result; } catch (const std::exception& e) diff --git a/src/openpose/wrapper/CMakeLists.txt b/src/openpose/wrapper/CMakeLists.txt index 73c3cc48..83c6de84 100644 --- a/src/openpose/wrapper/CMakeLists.txt +++ b/src/openpose/wrapper/CMakeLists.txt @@ -16,8 +16,8 @@ set(SOURCES_OPENPOSE ${SOURCES_OPENPOSE} ${SOURCES_OP_WRAPPER_WITH_CP} PARENT_SC if (UNIX OR APPLE) add_library(openpose_wrapper ${SOURCES_OP_WRAPPER}) - target_link_libraries(openpose_wrapper openpose_thread openpose_pose openpose_hand - openpose_core openpose_face openpose_filestream openpose_gui openpose_producer + target_link_libraries(openpose_wrapper openpose_thread openpose_pose openpose_hand + openpose_core openpose_face openpose_filestream openpose_gui openpose_producer openpose_utilities) install(TARGETS openpose_wrapper